What is Inferact?
Inferact is a scheduling and orchestration layer that runs above your inference engines (vLLM, TGI, or custom runtimes) and manages request routing, KV cache eviction, and batch formation across a heterogeneous fleet of GPU nodes.
It is designed for teams running open-weight models (Llama 3, Mistral, Mixtral, and similar) on fleets that mix accelerator generations: A100 and H100 servers alongside prosumer RTX hardware, or fleets where different nodes run different quantization formats.
Core capabilities
Route requests to nodes based on their memory bandwidth profile, current KV cache occupancy, and quantization format. A 7B INT4 request goes to the node best suited for it, not the next one in round-robin order.
System prompts shared across many requests are pinned in the KV cache. Eviction targets the least-reused suffixes first. In our internal benchmarks on system-prompt-heavy workloads, this reduces redundant prefill by up to 60%.
Requests join running batches at any decode step. The batch size adapts to per-node memory pressure. Speculative decoding is isolated to nodes with verified draft-model capacity.
When a node approaches OOM, the scheduler migrates queued requests to alternate nodes before the engine panics. In-flight requests are completed; new requests are not accepted until memory pressure normalizes.
How it fits into your stack
Inferact does not replace your inference engine. It wraps it. The standard deployment puts Inferact between your load balancer and your vLLM or TGI instances:
client requests
|
load balancer
|
iac-scheduler ← request routing, KV policy, batch control
/ | \
vLLM vLLM TGI ← inference engines (per node)
A100 H100 RTX ← GPU hardware