Open-Weight Inference Optimization
Max throughput from every GPU in your inference fleet
Inferact tunes batching, KV cache, and scheduling per accelerator so your open-weight deployment stops leaving compute on the table.
How It Works
How Inferact works
Deploy your model, let Inferact profile your hardware, and serve at peak throughput automatically.
Deploy your open-weight model
Point Inferact at any Hugging Face compatible checkpoint or local model directory. Inferact handles model loading, sharding, and runtime initialization across your available GPUs.
Hardware profiling generates your config
Inferact benchmarks each GPU at startup, measuring real FLOPS, memory bandwidth, and interconnect throughput. The result is a tuned YAML config specific to your exact hardware combination.
Serve requests at peak throughput
The optimized scheduler runs continuous batching with prefix-aware KV cache reuse, allocating capacity across GPUs by actual capability. Every request benefits from hardware-specific dispatch decisions.
Configuration that fits your hardware, not a generic template
Most inference stacks ship a single set of defaults and call it production-ready. Those defaults were written for a specific GPU, a specific model size, a specific memory capacity. Your fleet is probably none of those things.
Inferact profiles each accelerator at startup and builds a configuration from measurement, not assumption. Batch sizes, KV cache allocation, tensor parallelism groupings, and speculative decoding thresholds are all set from observed hardware behavior.
The output is a transparent YAML file you can inspect, version, and override. No magic constants hiding in a binary config.
- Measured FLOPS and memory bandwidth per device
- Per-GPU KV cache block allocation
- Heterogeneous tensor parallel groupings
- Continuous batching thresholds per memory tier
# Auto-generated from hardware profile on 2026-07-08
model: meta-llama/Meta-Llama-3-70B-Instruct
gpu_memory_fraction: 0.92
batching_strategy: "continuous"
kv_cache_policy: "prefix-aware"
max_num_seqs: 256
tensor_parallel_size: 4
speculative_decoding: true
oom_recovery: "drain-and-reroute"
# quantization detected from checkpoint dtype
quantization: "int8"Core Engine
Built for heterogeneous GPU fleets
Six scheduling and memory primitives designed to extract maximum throughput from mixed hardware deployments.
Continuous Batching
Requests join and leave the batch mid-flight without stopping the inference engine. KV cache slots are reclaimed immediately when sequences complete, keeping GPU utilization high during variable-length workloads.
Prefix-Aware KV Cache
Shared prompt prefixes are cached once in GPU memory and reused across all concurrent requests that share that prefix. Cache eviction policy is configurable per memory tier and GPU capacity profile.
Heterogeneous Fleet Scheduling
Inferact allocates workloads across mixed GPU hardware by measured throughput capacity, not nominal VRAM. An A6000 Ada and an A100 serving the same model receive requests proportional to their actual output rates.
Speculative Decoding
A smaller draft model predicts multiple tokens ahead; the target model validates them in a single forward pass. Acceptance rate is tracked per hardware configuration and the speculation window adjusts dynamically.
Quantization-Aware Dispatch
Inferact selects batch sizes and memory layouts based on quantization format detected from the checkpoint. int4 and int8 checkpoints receive dispatch parameters calibrated to their actual memory and compute tradeoffs.
OOM Recovery
Memory pressure is monitored per device on every scheduling cycle. When a GPU approaches its KV cache limit, Inferact drains in-flight sequences from that device and routes new requests to devices with available headroom.
Early Access
Join the early-access program
We are working with a small group of infrastructure teams running open-weight models in production or staging environments. Early-access participants get direct access to the Inferact engineering team, priority support, and influence over roadmap priorities.
If your team runs Llama, Mistral, Qwen, or other open-weight checkpoints on owned or leased GPU hardware, we want to hear about your throughput and memory management challenges.
The team
Start serving at peak throughput
Apply for early access or read the technical documentation to learn how Inferact fits your hardware stack.