vLLM, TGI, and Inferact: Throughput Benchmark on Llama-3 70B

Benchmarks of inference systems are easy to manipulate, usually accidentally. The operator makes a few reasonable choices about batch size, prompt length, and measurement window, and ends up with numbers that are highly specific to those choices without necessarily knowing it. This post is an attempt to be honest about what our benchmark actually measured, where the numbers were surprisingly close, and where Inferact showed a real difference.

All figures in this post are from our internal benchmarks on hardware we control. We are not claiming these results would hold on independent hardware with different tuning. The goal is transparency about methodology, not a marketing comparison.

Test environment

We ran this on a 4-node cluster, each node an 8xA100 80GB SXM4 machine with NVLink within each node. Nodes were connected via 400Gbps InfiniBand. Model: Llama-3 70B in BF16, tensor-parallel across 8 GPUs per node, no pipeline parallelism. Each system received the same traffic stream: a synthetic workload derived from ShareGPT prompt length distributions, input lengths between 512 and 2048 tokens, output lengths between 128 and 512 tokens, drawn from a Poisson arrival process at several request rates.

vLLM version 0.4.2 with PagedAttention, no quantization, default continuous batching. TGI 2.1.0 with continuous batching and the default Flash Attention 2 backend. Inferact ran its standard multi-node scheduler with continuous batching and its own KV cache management layer on top of the same attention backend. All three systems used the same Flash Attention 2 CUDA kernels for the actual attention computation.

We measured tokens per second (both input and output) and normalized by node count to get per-node throughput. We also measured p50 and p99 inter-token latency at several load levels. The benchmark ran for 30 minutes at each load level after a 5-minute warmup period.

What the numbers showed

At low to moderate load (under 60% of peak throughput), all three systems produced similar per-node throughput, within roughly 5% of each other. At this load level, the attention computation dominates and the scheduling overhead is small. There is no meaningful difference to report here, and we will not claim one.

At high load (80-95% of peak throughput), differences emerged. In our internal benchmarks, Inferact showed roughly 12-18% higher sustained throughput compared to a single-node vLLM deployment across all 4 nodes. Compared to TGI with its default configuration, the difference was slightly larger, around 15-22%. These numbers are sensitive to the specific request rate and prompt length distribution we used. With a uniform prompt length distribution (all inputs 512 tokens, all outputs 256 tokens), the gap narrows considerably.

The p99 latency difference was more pronounced than the throughput difference. At 90% of peak load, vLLM's p99 inter-token latency in our benchmark was approximately 2.3x its p50 latency. Inferact's p99 was approximately 1.6x its p50. TGI sat between the two, closer to vLLM. We attribute most of this p99 compression to the cross-node queue management in the Inferact scheduler, which we describe below.

What the numbers obscure

Single-node vLLM and TGI are not architected to coordinate across nodes. When you run 4 instances of vLLM or TGI and put a load balancer in front of them, each instance manages its own KV cache and batch queue independently. Under bursty traffic, one node can fill its KV cache while another has free VRAM, with no mechanism to redistribute load. The load balancer sees only request rates, not memory pressure.

This means our benchmark was not comparing vLLM-the-attention-kernel to Inferact-the-attention-kernel. Those are the same CUDA code. We were comparing vLLM-plus-4-independent-schedulers against Inferact-with-a-coordinated-scheduler. The throughput difference comes almost entirely from the scheduler layer, not from anything specific to our attention implementation.

If you run a single-node vLLM deployment and compare it against a single-node Inferact deployment on the same hardware, you will see smaller differences, mostly in KV cache utilization efficiency under bursty load rather than raw token generation throughput.

Where the Inferact scheduler adds throughput

The coordinated scheduler gives Inferact three capabilities that direct vLLM or TGI do not have in a multi-node configuration.

First: cross-node KV cache visibility. The scheduler knows, in real time, the KV cache occupancy of every node. When a new request arrives, routing considers available KV cache pages, not just current queue depth. Under high load with heterogeneous prompt lengths, this prevents the scenario where short-prompt requests pile up on a node whose cache is fragmented by partially evicted long-prompt sequences.

Second: cross-node prefix sharing. When two requests share a common prefix (same system prompt, for example), the scheduler can route them to the same node, allowing the KV cache pages for that prefix to be shared. vLLM has prefix caching within a single node. The Inferact scheduler extends this to the multi-node case, reducing redundant prefill computation for shared prefixes in high-concurrency scenarios.

Third: load-aware preemption. When a node's KV cache fills and it would otherwise evict in-flight sequence pages, the Inferact scheduler can migrate the pending (not yet scheduled) portion of the batch to a less-loaded node before eviction occurs. This is not zero-cost, but eviction and re-prefill from scratch is worse. The preemption path fires infrequently in steady state; its primary effect is on p99 latency under traffic spikes.

Caveats and honest limits

These results are specific to the synthetic workload we used. A uniform workload where all prompts have the same length will show smaller differences, because the scheduling decisions become simpler and cross-node coordination adds overhead without proportional benefit. If your production traffic is highly uniform, the incremental value of a coordinated scheduler over direct vLLM is smaller.

The benchmark did not test quantized models. INT4 inference changes the memory bandwidth profile significantly and may shift which system performs better on which metric. We have done separate profiling on quantized workloads but those numbers are not included here.

TGI's performance in our benchmark was notably sensitive to the max-batch-total-tokens configuration. With default settings, TGI under-batched on long-context requests in our workload. A TGI operator who had tuned this parameter for their specific prompt distribution would likely see better numbers than we report here.

Conclusions from the exercise

Multi-node LLM inference throughput benchmarks mostly measure scheduling and memory management quality, not kernel performance. All three systems use essentially the same attention kernels. The 12-22% throughput improvement we measured in our internal benchmarks on this specific workload reflects the coordinated scheduler, not superior CUDA code.

Whether that improvement is worth adopting a new system depends on your workload characteristics. High-concurrency deployments with bursty traffic and mixed prompt lengths are where the difference shows up most clearly. Low-concurrency or uniform-workload deployments will see smaller gains. We think that is the honest framing of what these numbers mean.