API Reference

Configuration schema

The Inferact scheduler reads a YAML config file at startup. The full schema is documented below with all accepted keys and their types.

Top-level structure

scheduler:
  port: 9000               # int, default 9000
  host: "0.0.0.0"         # string, default "0.0.0.0"
  log_level: info          # debug | info | warn | error
  request_timeout_ms: 30000

nodes:
  - id: string             # required, unique identifier
    hardware: string       # a100-80g | a100-40g | h100-80g | rtx-4090 | a6000 | custom
    backend: string        # vllm | tgi | custom
    endpoint: string       # http(s) URL of the inference engine
    models: list[string]   # model IDs this node can serve
    weight: float          # optional, 0.0-1.0, routing weight multiplier

kv_cache:
  policy: string           # lru | prefix-aware | adaptive
  pin_system_prompts: bool # default true
  eviction_threshold_pct: int  # 0-100, trigger eviction when KV used exceeds this

batching:
  strategy: string         # continuous | static
  max_batch_size: int      # per-node cap; omit to let the engine decide
  prefill_decode_split: bool  # route prefill and decode to separate node pools

Hardware identifiers

The hardware field tells the scheduler which memory bandwidth and compute profile to use when estimating routing cost. Use custom for hardware not in the list and provide a hardware_profile block.

nodes:
  - id: my-node
    hardware: custom
    hardware_profile:
      memory_bandwidth_gbps: 900
      vram_gb: 48
      nvlink: false

REST endpoints

POST /v1/chat/completions

OpenAI-compatible chat completions endpoint. The scheduler selects the target node and proxies the request. All standard OpenAI request fields are forwarded verbatim; routing extensions are in the x-inferact namespace.

{
  "model": "meta-llama/Llama-3-8B-Instruct",
  "messages": [...],
  "x-inferact": {
    "node_affinity": "node-a100-01",   // optional: pin to a specific node
    "kv_prefix_hint": "sys_v3",        // optional: cache key hint for this system prompt
    "priority": 1                      // optional: 0 (low) to 10 (high), default 5
  }
}

GET /status

Returns current state of all registered nodes including KV cache utilization and in-flight request counts.

{
  "scheduler_version": "0.9.4",
  "uptime_seconds": 3842,
  "nodes": [
    {
      "id": "node-a100-01",
      "hardware": "a100-80g",
      "backend": "vllm",
      "status": "healthy",       // healthy | degraded | offline
      "kv_cache_used_pct": 42.1,
      "requests_in_flight": 3,
      "requests_total": 1240,
      "p99_latency_ms": 820
    }
  ],
  "total_requests_routed": 4821
}

POST /admin/nodes/{id}/drain

Stops routing new requests to a node. In-flight requests complete normally. Use before node maintenance. Requires the admin API key in the X-Inferact-Admin-Key header.

POST /admin/nodes/{id}/restore

Re-enables routing to a previously drained node.

GET /metrics

Prometheus-format metrics. Scrape this endpoint with your monitoring stack.

Routing policies

KV cache policies

The kv_cache.policy field controls which requests are preferred for KV cache reuse:

lru: evict least recently used entries. Correct default for uniform traffic; no routing preference.
prefix-aware: pin entries that share a common prefix (system prompt). Route requests to the node that already has the prefix loaded. Reduces prefill cost on workloads with high system-prompt sharing.
adaptive: switch between lru and prefix-aware based on measured prefix hit rate. Activates prefix-aware mode when hit rate exceeds 40% over a rolling 5-minute window.

Batching strategies

The batching.strategy field controls how requests are grouped per node:

continuous: requests join an in-progress batch at any decode step. Maximizes throughput on long-running generation workloads.
static: requests wait for a full batch before any decode starts. Lower variance; useful for batch processing pipelines where latency is not time-sensitive.

Metrics reference

All metrics are prefixed with inferact_.

# HELP inferact_requests_total Requests routed since scheduler start
inferact_requests_total{node="node-a100-01",model="Llama-3-8B"} 1240

# HELP inferact_kv_cache_used_ratio KV cache utilization (0.0-1.0)
inferact_kv_cache_used_ratio{node="node-a100-01"} 0.421

# HELP inferact_prefix_hit_ratio Prefix cache hit rate (rolling 5m)
inferact_prefix_hit_ratio{node="node-a100-01"} 0.63

# HELP inferact_request_latency_seconds Request latency histogram
inferact_request_latency_seconds_bucket{node="node-a100-01",le="0.5"} 820
inferact_request_latency_seconds_bucket{node="node-a100-01",le="1.0"} 950
inferact_request_latency_seconds_bucket{node="node-a100-01",le="+Inf"} 1240

# HELP inferact_oom_events_total OOM events recovered by the scheduler
inferact_oom_events_total{node="node-rtx-01"} 2