Speculative Decoding and Batching: Why They Conflict and How to Manage It

Speculative decoding is one of the more interesting latency optimizations in LLM inference right now. The basic mechanism: a small draft model generates K candidate tokens, then the larger target model verifies all of them in a single forward pass. If the target agrees with the draft's predictions (checked probabilistically), you get K tokens for roughly the cost of one target-model forward pass. On workloads where the draft model tracks the target model's distribution closely, this produces 2-3x wall-clock latency improvement for individual requests.

The problem is that this improvement is fragile under batching. Most production setups are not running batch size 1. Understanding why the gains degrade at larger batches, and how to keep speculative decoding useful without letting it hurt throughput on non-speculative requests, is a problem we have spent real time on at Inferact.

Why batch size kills speculative decoding efficiency

The mechanics require the draft model to run on the same batch as the target model. For each sequence in the batch, the draft model generates K speculative tokens, then the target model verifies all of them in one pass. The speedup depends on verification being parallelizable: the target can check K tokens in roughly the same time it takes to generate 1.

This breaks down as batch size grows for several connected reasons.

First, the draft model forward pass grows with batch size. A larger batch means the draft model takes longer to generate its K candidates per sequence. At small batches (1-4 sequences), the draft model is fast relative to the target's verification pass. At batch size 32 or above, the draft model's latency starts to dominate, and the arithmetic of "cheap draft plus cheap verification" no longer holds.

Second, KV cache pressure on the draft model increases. The draft model maintains its own KV cache. With a large batch of diverse requests, that cache fills with varied prefixes that have low overlap. Eviction rates increase. The draft model runs on partially evicted context, which degrades prediction quality. Measured acceptance rate drops even before you hit compute saturation on the draft model.

Third, acceptance rate degrades at high batch diversity. This is empirical rather than analytical, but we see it consistently: when short-generation and long-generation requests are mixed in the same speculative batch, the draft model's per-sequence prediction quality drops for the long-generation requests. The draft is being asked to maintain coherent per-sequence state across very different sequence lengths simultaneously.

The combined effect on our A100 nodes, using a 7B draft model with Llama-3 70B as the target: measured speedup drops from roughly 2.1x at batch size 4 to near 1.0x at batch size 32. At batch size 32 you are running a draft model on every request and getting nothing for it.

The fleet-level angle

A single-server perspective on this problem leads to a binary choice: enable speculative decoding or not. With a fleet of nodes, you have a third option: designate specific nodes for speculative decoding and route only the requests that benefit from it to those nodes.

In the Inferact scheduler, we track per-node "speculative capacity." Each node that has a loaded draft model reports its current batch occupancy and a rolling acceptance rate estimate derived from recent request history on that node. The scheduler uses both signals for routing decisions.

A request is routed to a speculative node if all of the following hold: the node has draft model weights loaded, current batch size is below a configured ceiling (we use 8 as the default), and the recent acceptance rate for this request's prompt category is above 0.6. If any condition is not met, the request routes to a standard node. The ceiling and acceptance rate threshold are configurable per-deployment; the defaults reflect our profiling on 7B-draft plus 70B-target on A100.

What happens at the boundary

The interesting operational case is when a speculative node is near its batch size ceiling and a burst of incoming requests arrives. We do not force a mid-batch reconfiguration: evicting the draft model or re-routing in-flight requests is expensive and introduces latency spikes worse than the problem being solved. Instead, the scheduler marks that node as "speculative-saturated" and stops routing new speculative-eligible requests to it until batch occupancy falls back below the threshold.

Some requests that would benefit from speculative decoding end up on standard nodes during saturation events. This is a small efficiency loss. The alternative is degraded performance for all active sequences on the speculative node, which is worse by a larger margin. The scheduler polls per-node state on a 100ms interval by default, so the speculative node re-enters the eligible pool promptly once it drains.

One thing to monitor: if your speculative nodes are hitting the saturation threshold constantly, the root cause is almost always an undersized ceiling for your actual traffic patterns, not a fundamental problem with the approach. Raising the ceiling to 12 or 16 and re-profiling the acceptance rate at those batch sizes is the right diagnostic step before declaring speculative decoding impractical for your workload.

Draft model selection matters more than most teams realize

The acceptance rate is the variable that controls how useful speculative decoding is, and acceptance rate is largely determined by how well the draft model's token distribution matches the target model's distribution on your specific prompt patterns.

A generic 1B or 3B draft model will produce lower acceptance rates than a domain-specific draft model fine-tuned on your actual prompt distribution. The difference is significant: we have seen acceptance rates move from 0.45 on a generic draft to 0.72 on a domain-fine-tuned draft on the same target model and workload. At 0.45, speculative decoding at batch size 4 barely breaks even. At 0.72, batch size 4 gives roughly 1.8x speedup.

The practical implication: if you are measuring speculative decoding with a stock draft model and seeing marginal gains, the problem may not be your infrastructure configuration. It may be draft model selection. Fine-tuning a small model on your domain costs less than most teams expect and pays off directly in acceptance rate.

When speculative decoding is not the right tool

We are not claiming speculative decoding is universally beneficial. It works well for long-form generation workloads where the draft model tracks the target distribution closely: code generation with a code-specific draft, structured output generation where the output format is predictable, or document expansion where the next few words are highly constrained by the preceding text.

It works poorly for short-output workloads (under 64 tokens), highly diverse prompt mixes with no dominant generation pattern, and workloads where your binding constraint is throughput rather than per-request latency. If your SLO is tokens per second per GPU rather than p50 TTFT, continuous batching with a large batch size on standard nodes will outperform speculative decoding on dedicated smaller-batch nodes. The right configuration depends on what you are actually measuring.

There is also a hardware cost consideration. Running a draft model requires VRAM that cannot be used for the target model or for KV cache pages. On 80GB A100 nodes, loading a 7B draft model in BF16 uses roughly 14GB, reducing the VRAM available for target model inference. This cost is worth paying when acceptance rate is high and latency is the binding SLO. When acceptance rate is low or when throughput is the primary goal, that 14GB is better used for KV cache capacity.

Configuration notes for fleet operators

Keep speculative decoding on designated nodes rather than enabling it fleet-wide. This lets you tune the batch size ceiling and draft model independently per node type without affecting the rest of the fleet. It also makes it easier to measure the actual impact: compare TTFT distributions between requests routed to speculative nodes and requests routed to standard nodes, with the same model and workload.

Monitor acceptance rate per prompt category, not just as a global average. A 0.65 fleet-average acceptance rate can mask a 0.85 for code requests and a 0.40 for open-ended chat requests. The 0.40 segment is running draft model inference and getting minimal benefit. Separating routing by prompt category, or simply identifying which categories drive the low acceptance rate and excluding them from speculative routing, will improve the overall efficiency of the speculative nodes.

The Inferact scheduler exposes per-node acceptance rate, per-node batch saturation events, and per-category acceptance rate breakdowns. These are the primary signals for deciding whether your speculative decoding deployment is working as intended or simply adding overhead.