Berkeley, CA / Founded 2023
Built by the people who ran inference at scale
We spent years debugging OOM events at 3am, watching GPU utilization stay at 40% during load spikes, and hand-tuning batch sizes for mixed accelerator clusters. Inferact is the tool we wished existed.
Our focus
Why inference scheduling is still unsolved
The hardware problem
Consumer and prosumer GPUs are now capable of running 7B to 70B models. An RTX 4090 cluster can produce substantial throughput at a fraction of the cost of a hyperscaler. The problem is that mixing A100s, A6000s, and RTX cards in the same fleet breaks every scheduler that assumes uniform hardware. Different memory bandwidth profiles, different compute densities, and different NVLink vs PCIe topologies mean a one-size batch size is always wrong for some device in the fleet.
The software gap
vLLM and TGI are excellent single-node engines. They were not designed for multi-node heterogeneous fleets with different quantization formats per node. KV cache eviction in both is LRU by default, which is correct for uniform traffic but wastes memory when system prompts are shared across 80% of requests. Inferact sits above the inference engine layer, orchestrating across nodes with the context that a single engine lacks.
Team
The people building it
Has spent years working on the scheduling and memory management layers that make inference on mixed GPU fleets tractable. Cares about the gap between what GPU hardware can theoretically do and what production workloads actually extract.
Focuses on CUDA kernel profiling, KV cache memory policy design, and the low-level work that determines whether a scheduler suggestion is theoretically correct or actually faster. Built paged attention extensions while contributing to open-source inference tooling before co-founding Inferact.
Systems programmer with a background in distributed job scheduling and hardware abstraction. Owns the multi-node tensor parallelism layer and the fault-tolerance paths that let the scheduler recover from OOM events without dropping in-flight requests.
Values
How we work
Measurement over intuition
Every scheduler change is validated against a synthetic benchmark before touching a production fleet. Intuitions about GPU behavior are frequently wrong; numbers are not.
Build for real hardware
We test on the hardware our users actually run. RTX 4090s, A100 80G servers, and consumer-grade mixed clusters are first-class targets, not afterthoughts.
Transparent internals
We write about what we learn: scheduling algorithms, KV cache eviction trade-offs, and quantization interaction effects. The blog is where we think in public.
Small team, large surface
Three engineers covering the full stack from CUDA kernel profiling to Kubernetes Helm chart packaging. We stay small on purpose; every person who joins needs to move the scheduler forward.
We are looking for early access partners
If you run a heterogeneous GPU fleet serving open-weight models and want to work directly with us on scheduling problems, get in touch.