The conventional wisdom in LLM inference has converged strongly on continuous batching. If you read the major inference framework documentation, continuous batching is presented as the approach that maximizes throughput by eliminating the GPU idle time that static batching wastes while the slowest sequence in a batch finishes. This is mostly correct, but the framing elides the cases where static batching is actually preferable. This post is about those cases and how we decide between them in Inferact's scheduler.
We are not going to rehash the basic mechanics of how each approach works. If you need that background, the original Orca paper and the vLLM blog post cover it well. What we want to focus on is the specific conditions where the choice matters and what the latency distributions look like in each regime.
The Throughput Argument for Continuous Batching
Static batching pads every sequence in a batch to the maximum sequence length, then runs the forward pass over that fixed-size batch until all sequences are complete. Sequences that finish early stall GPU utilization while waiting for the longest sequence in the batch. On workloads with highly variable output lengths, the wasted compute time is significant. If your output length distribution has a p50 of 200 tokens and a p99 of 2000 tokens, a static batch of 32 sequences spends most of its GPU time generating tokens for the 1 percent of long-tail sequences while the other 31 have already finished.
Continuous batching solves this by inserting new requests into the batch as slots free up. When a sequence completes, its KV cache pages are released and a waiting request from the queue is immediately promoted into the batch, with its prefill phase executed at the next scheduling iteration. GPU utilization stays high because there is almost always a new sequence ready to fill the slot left by a completed one.
In our internal benchmarks on Llama-3 8B at 100 requests per second with output lengths drawn from a distribution matching typical chat workloads (roughly lognormal with median 150 tokens, mean 320 tokens), continuous batching produced approximately 2.3x higher tokens-per-second throughput compared to static batching with a batch size tuned for the same hardware.
The Latency Argument for Static Batching
Here is what the throughput framing misses: continuous batching introduces non-determinism in per-request latency that static batching does not have.
In a continuous batching system, a new request that arrives when the batch is full must wait in the queue until a slot opens. That wait time depends on when the currently running sequences complete, which in turn depends on their output lengths, which the scheduler does not know in advance. A request that arrives just as a batch of unusually long sequences starts running may wait for the entire duration of those long sequences before it can even begin prefill.
This creates tail latency behavior that is qualitatively different from static batching. In static batching with a fixed batch size, the maximum queue wait time for a new request is bounded by the longest possible batch execution time. In continuous batching, a queue of 50 requests waiting for 10 open slots creates a much more complex waiting-time distribution, and the p99 TTFT can be substantially higher than the mean TTFT during traffic spikes.
In our internal testing on Llama-3 70B at an A100 node loaded to 80 percent of throughput capacity, continuous batching produced a mean TTFT of approximately 820 ms and a p99 TTFT of approximately 4.1 seconds. Static batching (batch size 16) at the same load produced mean TTFT of 1.1 seconds and p99 TTFT of 2.3 seconds. The mean is worse for static batching, but the tail is significantly better.
Which Metric Governs Your Decision
The core question is whether your application is more sensitive to average throughput or to p99 latency. These are genuinely different optimization objectives, and the framework choice should follow from which one you actually need.
Applications where continuous batching is clearly better: offline document processing, batch embeddings, any workload where you are generating large volumes of text and measuring total throughput rather than per-request latency. The person on the other side of the request is not a human waiting for a response, so tail latency variation does not matter.
Applications where static batching deserves consideration: interactive chat interfaces where users will notice a 4-second first-token delay. Voice-to-text pipelines where latency above 500 ms breaks the user experience. Real-time code completion where a p99 TTFT above 2 seconds makes the product feel broken. These are latency-sensitive workloads where predictable tail behavior matters more than peak throughput.
We are not saying static batching is generally preferable. On most realistic workloads, continuous batching wins on the metric that matters most (throughput, and usually mean latency too). The argument is that the decision should follow from a measurement of your actual p99 TTFT targets against your actual traffic pattern, not from a default preference for the newer approach.
How Traffic Patterns Interact with the Choice
The advantage of continuous batching degrades as concurrency drops. At low concurrency (single-digit active sequences), a static batch and a continuous batch look nearly identical because there are no queued requests waiting for slots. The throughput gains of continuous batching only manifest when the queue is full enough that freed slots are immediately refilled.
Conversely, the tail latency disadvantage of continuous batching becomes more pronounced as concurrency rises. Under heavy load, the queue grows, slot availability becomes uncertain, and high-TTFT outliers accumulate. The latency distribution widens in proportion to the ratio of mean queue depth to batch size.
On Llama-3 8B, we see the inflection point in our testing at approximately 40-50 concurrent sequences. Below that, continuous and static batching produce broadly similar p99 TTFT values. Above it, the gap widens in the direction that matters for your use case: static batching produces a tighter distribution, continuous batching produces higher peak throughput.
Hybrid Approaches and What We Actually Do
In practice, Inferact's scheduler does not make a binary choice between pure continuous and pure static batching. We use a hybrid: continuous batching is the default for all requests, but we impose a maximum queue wait time threshold (currently configurable, defaulting to 1.5x the median TTFT observed over the last 60 seconds). If a request has been in the queue longer than that threshold, it gets priority promotion and will preempt the oldest non-prefill slot in the current batch.
This is not the same as static batching, but it provides a weaker form of the tail latency guarantee that static batching offers. The tradeoff is that preemption has a cost: the evicted sequence loses its decode progress and must be rescheduled, which wastes some compute. We calibrate the threshold so that preemption happens rarely (less than 2 percent of sequences in our test workloads) while still preventing extreme tail latency events.
For teams that do not want to tune this parameter, the simpler recommendation is: use continuous batching for throughput-sensitive deployments, use static batching with a conservative batch size for latency-sensitive deployments, and measure your actual p99 TTFT under realistic load before deciding that continuous batching is categorically better for your use case.
Llama-3 70B: Where the Gap Is Larger
The 70B model changes the calculus somewhat. At 70B, the per-forward-pass compute time is substantially higher, which means that long sequences in a batch hold the GPU for longer before completing. This amplifies both the throughput waste in static batching and the tail latency problem in continuous batching.
In our internal measurements on Llama-3 70B (4-GPU TP on A100 SXM4), continuous batching at 80 percent capacity produced roughly 3.1x higher tokens-per-second compared to static batching, a larger gap than the 8B results. The p99 TTFT under continuous batching was also worse in absolute terms: approximately 7.8 seconds versus 3.2 seconds for static batching at the same load. Both the upside and the downside are larger at 70B.
The 70B case illustrates the core tension clearly: if your throughput constraint is binding (you need every token per second you can get), continuous batching is the only viable choice. If your latency constraint is binding (p99 TTFT must stay under some threshold), static batching with a smaller batch size is a legitimate option, at the cost of significantly lower throughput at high load.