Platform Architecture
Built around how GPUs actually run inference
Every scheduling decision in Inferact is shaped by the hardware it runs on: memory bandwidth profile, compute density, PCIe vs NVLink topology, and current batch pressure.
Core Capabilities
Five layers that compose into one runtime
Each layer is independently tunable but designed to reinforce the others at high concurrency.
Continuous Batching Engine
Requests enter and leave mid-batch without stalling inflight iterations. The scheduler operates at the iteration level, not the batch level, so a short prompt never waits behind a 4096-token generation.
Prefix-Aware KV Cache
Shared prompt prefixes are computed once and stored across requests. Inferact tracks prefix trees per model and evicts based on reuse probability, not simple LRU. In our internal testing on system-prompt-heavy workloads, this reduces redundant prefill by 40-70% depending on prefix sharing rate.
Quantization-Aware Dispatch
INT4 and FP16 shards have different throughput characteristics. Inferact profiles each GPU at startup and routes requests to the shard whose weight format matches the current batch composition for maximum tokens-per-second.
Speculative Decoding Integration
Speculative decoding accelerates low-concurrency requests but competes with batching at high load. Inferact selects the draft model window dynamically based on current queue depth and acceptance rate history per model.
OOM Recovery and Pressure Guard
Memory pressure signals from CUDA are intercepted before OOM events. Inferact preempts lower-priority requests and checkpoints KV blocks to CPU memory, recovering in-flight capacity without crashing the worker process.
Real-Time Metrics Plane
Token throughput, queue depth, p50 and p99 latency, and KV cache hit rate are emitted every 250ms over a Prometheus-compatible endpoint. The scheduler reads its own metrics to adjust batch targets in a closed feedback loop.
Technical Architecture
How the runtime is structured
Request intake and priority queue
Incoming requests are tokenized and assigned a priority score based on SLA tier, queue age, and prefix cache availability. A weighted fair-queuing algorithm prevents head-of-line blocking while ensuring high-priority traffic is served within its latency budget.
def score_request(req, now):
age = now - req.enqueue_ts
prefix_hit = check_kv_cache(req.tokens)
return req.sla_weight * age
+ prefix_hit * 0.4
Per-GPU worker model
Each accelerator runs an isolated worker process. Workers report their free KV blocks and compute capacity to the central scheduler every 50ms. The scheduler never issues a batch that would exceed a worker's current memory headroom, avoiding the cascade that leads to OOM.
@dataclass
class WorkerState:
gpu_id: str
free_kv_blocks: int
active_seqs: int
util_pct: float
Multi-node tensor parallelism routing
For models too large for a single node, Inferact distributes tensor-parallel shards across nodes and routes attention-layer communication over InfiniBand or RoCE. Pipeline parallelism depth is configured per-model based on measured NVLink vs. cross-node bandwidth ratios.
Deployment Model
Self-hosted on your infrastructure
Inferact runs as a sidecar to your existing inference stack. No cloud dependency, no data leaves your cluster.
A single pip install deploys the Inferact scheduler daemon alongside your CUDA drivers. Supports CUDA 11.8+ and Python 3.10+. Kubernetes Helm chart available in the quickstart.
Drop a YAML config listing your GPU nodes, model shards, and quantization types. Inferact profiles each accelerator on first startup and sets per-node batch ceilings automatically.
An OpenAI-compatible REST endpoint proxies all traffic through the Inferact scheduler. Drop-in compatible with any client that targets the completions or chat API.
Prometheus metrics expose tokens/second, queue depth, KV cache hit rate, and p99 latency. The feedback loop tightens batch targets as the system learns each GPU's steady-state profile.
Ready to stop leaving tokens on the table?
Request early access and we will walk you through a hardware profile of your fleet.