Quickstart

Prerequisites

One or more GPU nodes running vLLM 0.4+ or TGI 2.0+
Python 3.10+ on the scheduler host
Network access between scheduler host and GPU nodes (port 8080 by default)
An Inferact early access token (request one at contact page)

Step 1: Install the scheduler

pip install inferact-scheduler

inferact --version
# inferact-scheduler 0.9.4

Step 2: Write a fleet config

Create a YAML file describing your nodes. Each node entry specifies the hardware type, the backend engine, and the endpoint the scheduler will call.

scheduler:
  port: 9000
  log_level: info

nodes:
  - id: node-a100-01
    hardware: a100-80g
    backend: vllm
    endpoint: http://10.0.1.11:8080
    models:
      - meta-llama/Llama-3-8B-Instruct
      - meta-llama/Llama-3-70B-Instruct

  - id: node-rtx-01
    hardware: rtx-4090
    backend: vllm
    endpoint: http://10.0.1.21:8080
    models:
      - meta-llama/Llama-3-8B-Instruct

kv_cache:
  policy: prefix-aware
  pin_system_prompts: true
  eviction_threshold_pct: 85

Step 3: Start the scheduler

inferact serve --config fleet.yaml --token $IAC_TOKEN

# Inferact scheduler starting...
# Loaded 2 nodes (1 a100-80g, 1 rtx-4090)
# KV policy: prefix-aware
# Listening on http://0.0.0.0:9000

Step 4: Send a request

Inferact exposes an OpenAI-compatible /v1/chat/completions endpoint. Point your client at the scheduler instead of directly at the engine.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:9000/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3-8B-Instruct",
    messages=[
        {"role": "user", "content": "Explain paged attention in two paragraphs."}
    ]
)

print(response.choices[0].message.content)

Step 5: Verify routing

The scheduler exposes a metrics endpoint at /metrics (Prometheus format) and a status endpoint at /status.

curl http://localhost:9000/status | python -m json.tool

# {
#   "nodes": [
#     {
#       "id": "node-a100-01",
#       "status": "healthy",
#       "kv_cache_used_pct": 42.1,
#       "requests_in_flight": 3
#     },
#     {
#       "id": "node-rtx-01",
#       "status": "healthy",
#       "kv_cache_used_pct": 18.7,
#       "requests_in_flight": 1
#     }
#   ],
#   "total_requests_routed": 4
# }