Prefill and decode are fundamentally different computations running on the same hardware. Prefill processes all input tokens in parallel: large matrix multiplications, high arithmetic intensity, GPU compute utilization running at 80-90% of theoretical peak. Decode generates one token per step per sequence: small matrix-vector products, low arithmetic intensity, GPU bound almost entirely by HBM read bandwidth. Running both phases on the same node means neither gets optimal hardware utilization. The node is over-provisioned for bandwidth during prefill and over-provisioned for compute during decode.
Prefill-decode disaggregation separates these phases: a prefill node handles the initial processing of the prompt, then transfers the resulting KV cache to a decode node that handles the rest of generation. The idea has been discussed in the research literature for a few years, but making it work in a production scheduler involves a set of implementation problems that are not obvious from the high-level description.
The utilization problem on colocated nodes
On a standard inference node running both prefill and decode simultaneously, the GPU's compute utilization profile oscillates. When a large batch of new requests arrives simultaneously, the node runs prefill for all of them, saturating compute. During the decode phase that follows, compute drops dramatically as the GPU repeatedly reads KV cache entries from HBM. Under continuous batching, new requests arrive asynchronously, so prefill and decode run interspersed. The GPU spends time in a mixed state that is suboptimal for both operations.
The practical consequence: a node running a 70B model under mixed workload typically shows 50-65% compute utilization and 60-75% HBM bandwidth utilization during steady-state inference, despite neither metric reaching its ceiling. The GPU is being asked to be good at two different things simultaneously and ends up adequately efficient at both rather than excellent at either.
Disaggregation trades this mixed efficiency for specialization. Prefill nodes run at high compute utilization, processing long input sequences in large batches. Decode nodes run at high HBM bandwidth utilization, generating output tokens for many sequences simultaneously. Both are closer to their respective hardware ceilings.
KV cache transfer: the hard part
After a prefill node completes the prompt processing, the KV cache for that sequence (keys and values for every transformer layer, accumulated for every input token) must be transferred to the decode node that will handle generation. This transfer is not free.
For Llama-3 70B in BF16, the KV cache per token is 2 bytes x 2 (key and value) x 8 (key-value heads per layer) x 128 (head dimension) x 80 (layers) = 327,680 bytes, roughly 320KB per token. For a 2048-token prompt, the KV cache transfer payload is approximately 640MB. Across an InfiniBand 400Gbps link, this transfer takes roughly 12.8ms. For a 256-token prompt, the transfer is 80MB and takes 1.6ms.
This transfer latency adds directly to the time-to-first-token (TTFT) for every request. A 12.8ms KV transfer on top of a 150ms prefill computation adds 8.5% to TTFT. Whether this is acceptable depends on your SLO. For batch inference workloads where TTFT is not a hard constraint, the KV transfer cost is easily justified by the throughput improvement on decode nodes. For interactive use cases with strict TTFT requirements, the transfer overhead needs careful accounting.
The KV cache for long-context requests is larger, but so is the prefill computation time, so the ratio of transfer overhead to prefill time stays roughly constant as context length grows. At 128K context, both prefill time and KV transfer scale proportionally to the context length, and the overhead fraction remains similar to shorter contexts.
Node ratio selection
The ratio of prefill nodes to decode nodes for a given workload determines whether one pool bottlenecks the other. If you have too many prefill nodes relative to decode nodes, the decode nodes become the throughput ceiling and prefill nodes sit idle waiting for decode capacity. Too many decode nodes, and decode capacity goes unused while prefill completes quickly.
The right ratio depends on the input-to-output token ratio of your workload. For a workload where average input is 1024 tokens and average output is 512 tokens (2:1 input/output ratio), prefill work is 2x the output work per token count. But prefill is compute-bound at high efficiency while decode is bandwidth-bound at different efficiency. The node ratio that balances these is not simply 2:1.
A rough empirical formula: measure the prefill throughput of a single prefill node (tokens per second) and the decode throughput of a single decode node (tokens per second, output only) on your target workload. Divide the decode throughput by the prefill throughput, then multiply by the expected output-to-input token ratio of your workload. This gives an approximate prefill-to-decode node ratio. For a 70B BF16 model on 8xA100 with 1024 input / 512 output distribution, we find ratios around 1:3 to 1:4 (one prefill node for every 3-4 decode nodes) work well. The exact ratio needs empirical tuning on your traffic distribution.
Scheduler implementation: the coordination problem
Disaggregation requires the scheduler to coordinate across two node pools. When a request arrives, the scheduler assigns it to a prefill node. When prefill completes, the scheduler must transfer the KV cache and assign the request to a decode node. This handoff introduces a state management problem: the scheduler must track where each in-flight request is in its lifecycle (waiting for prefill, in prefill, transferring KV, waiting for decode, in decode) and manage the assignment to each pool separately.
A secondary problem: if a decode node fails or becomes overloaded after a request's KV cache has been transferred, the KV cache cannot simply be moved to another decode node without an additional transfer operation. The scheduler must decide whether to pay the re-transfer cost or queue the request until the original decode node recovers.
In the Inferact implementation, each request carries a lifecycle state field that the scheduler updates atomically. The transfer operation is asynchronous: the prefill node pushes the KV cache payload to the scheduler's transfer buffer, and the scheduler dispatches it to the assigned decode node in the background. If the target decode node is unavailable at transfer time, the request queues in the transfer buffer rather than failing. This adds memory overhead (the transfer buffer holds KV payloads for in-flight transfers) but avoids request failures on transient decode node unavailability.
When disaggregation is not worth implementing
We are not suggesting every inference deployment should disaggregate prefill and decode. There are clear cases where the added complexity is not justified.
Small fleets (fewer than 4 total nodes) do not have enough nodes to create meaningful specialization. With 2 prefill and 2 decode nodes, the system is barely larger than a typical collocated deployment, and the coordination overhead eats the efficiency gains.
Workloads with very short inputs and outputs (under 256 tokens average for both) have low prefill-to-decode time ratios. The prefill phase completes so quickly that the GPU's compute underutilization during decode is a small fraction of total time. Disaggregation's benefit scales with the fraction of time spent in each phase; for short-context workloads, that fraction is too small to justify the architectural change.
Strictly latency-bound workloads where every millisecond of TTFT matters should measure the KV transfer overhead against their TTFT SLO before committing to disaggregation. For some use cases, an extra 10-15ms of TTFT from the transfer is unacceptable regardless of throughput improvement.
What the utilization numbers actually look like
On a test cluster we ran last quarter (8 nodes total, 2 prefill nodes and 6 decode nodes, Llama-3 70B BF16, average input 1536 tokens / average output 768 tokens), disaggregated prefill-decode showed the following versus collocated deployment on the same 8 nodes:
Prefill node compute utilization: 84% average versus 62% average on collocated nodes. Decode node HBM bandwidth utilization: 79% average versus 64% average on collocated nodes. Total system throughput (output tokens per second across all 8 nodes): approximately 23% higher for disaggregated. TTFT increased by 9ms on average due to KV transfer time.
These numbers are specific to this workload distribution and node count. Different input/output ratios or different node counts will produce different figures. But the direction of the change (higher utilization per hardware type, higher throughput, some TTFT cost) is consistent with the theoretical model.
Building a fleet large enough to disaggregate?
Inferact's scheduler supports prefill-decode disaggregation with asynchronous KV cache transfer and lifecycle-aware request routing. We work directly with early access partners on fleet-specific configuration.
Request Early Access