CUDA Graph Replay: Eliminating Kernel Launch Overhead for Inference

Every CUDA kernel launch carries CPU-side overhead: the driver validates arguments, sets up execution state, and submits the kernel to the GPU command queue. For a single kernel launch, this is microseconds. For a full Llama-3 70B forward pass, which involves hundreds of kernel launches per layer across 80 layers, that overhead accumulates into something measurable, particularly at the decode stage where forward passes are frequent and small.

CUDA graphs address this by recording all kernel launches for a computation and replaying the entire recorded sequence with a single API call. The GPU executes the same sequence of kernels without the CPU walking through the launch logic again. For inference, this translates directly into reduced per-token latency, most visibly at the decode phase.

Why decode phase benefits more than prefill

During prefill, the forward pass processes all input tokens in parallel. The batch is typically large (in token count), the kernels are long-running, and the kernel launch overhead is small relative to actual kernel execution time. The GPU is doing meaningful work for each launch.

Decode is different. Each decode step generates one token per sequence. The batch might be 32 sequences, but each sequence adds only 1 new token to its KV cache and runs a forward pass over essentially a single-token input with attention over the full cached context. The compute per kernel is low. The ratio of launch overhead to actual kernel work is much higher than during prefill. This is where CUDA graph replay produces the largest per-token speedup.

In our profiling on A100 with Llama-3 70B BF16, decode step latency dropped from approximately 28ms to approximately 21ms after enabling CUDA graph replay. That is a 25% improvement on the decode step specifically. Prefill latency was unchanged to within noise.

How graph capture works in practice

CUDA graph capture records the exact kernel sequence for a specific set of inputs. The critical constraint: the kernel sequence must be identical for capture and replay. This means the graph captures a specific batch size. If your production batch size varies (which it always does with continuous batching), you need to either capture one graph per distinct batch size, or fall back to eager execution for batch sizes that do not have a captured graph.

vLLM's CUDA graph implementation captures graphs for a set of batch sizes defined at initialization: by default, powers of two plus some intermediate values (1, 2, 4, 8, 16, 32, 48, 64...). For a decode step with batch size 12, it would replay the batch-size-16 graph with padding (12 real sequences, 4 padding sequences). The padded sequences execute but their outputs are discarded. The overhead of the padding depends on the ratio of pad to real sequences; at batch 12 out of 16 this is modest.

The set of captured batch sizes is a configuration decision. Capturing more batch sizes reduces padding waste but increases initialization time and CUDA memory used by the graph capture process. For most production deployments, a geometric series covering the expected batch size range is sufficient.

Graph capture failures and silent fallback

CUDA graph capture can fail for several reasons, and this is where things get operationally tricky. The most common failure modes we have seen:

Dynamic control flow within the model's forward pass. If the forward pass includes conditional logic that varies based on input content (not common in standard transformer inference, but it appears in some custom models and adapters), the kernel sequence changes between calls and capture fails or produces a graph that produces incorrect results.

CUDA operations that do not support graph capture. CUDA's graph API explicitly excludes certain operations including host-device synchronization points and some NCCL collective communication patterns. Tensor-parallel inference that uses certain NCCL operations may not be fully graphable without restructuring the communication pattern.

The dangerous case is silent fallback. Some inference frameworks will detect a graph capture failure and fall back to eager execution without surfacing an error. The operator sees slightly higher than expected decode latency, inspects the GPU utilization metrics (which look normal), and does not realize graph replay is not active. The way to verify: profile a decode step with Nsight Systems or torch.profiler and check whether you see cudaGraphLaunch in the trace or individual kernel launches. The trace tells you immediately which path is executing.

Memory overhead of captured graphs

Captured CUDA graphs consume GPU memory: the graph object itself stores the recorded kernel parameters, memory pointers, and execution state. For Llama-3 70B with graphs captured for 16 different batch sizes, the graph objects consume roughly 2-4GB of VRAM on A100. This is VRAM that cannot be used for KV cache pages.

The tradeoff: the KV cache pages displaced by graph memory reduce the maximum number of concurrent sequences you can serve before eviction starts. On an 80GB A100 node with a 70B model, 4GB of graph memory represents roughly 8-10% of the VRAM available for KV cache after model weights are loaded. Whether this is a good tradeoff depends on your workload: if you are latency-bound and your KV cache is not frequently pressured, the graph replay latency benefit outweighs the KV cache reduction. If you are heavily throughput-bound and your KV cache is under constant pressure, more KV cache pages may be worth more than the decode step speedup.

Integration with continuous batching schedulers

The interaction between CUDA graphs and continuous batching requires care. Continuous batching allows new sequences to enter and complete sequences to leave the batch between decode steps. This means the batch size changes dynamically. The scheduler needs to select which captured graph to use for each decode step, accounting for the current batch size and the nearest available captured graph size (choosing the next-larger captured size to accommodate all sequences with padding).

A subtle issue arises with the KV cache pointers stored inside the graph. A captured CUDA graph records specific GPU memory addresses. The KV cache pages for each sequence must reside at those addresses at replay time, or the graph will read from wrong memory. This means the KV cache allocator and the graph capture machinery need to be coordinated: page allocation must be stable within a decode step, and the graph must be parameterized over the KV cache addresses rather than having them baked into the captured state.

In vLLM, this is handled by using CUDA graph input tensors for the KV cache pointers, which allows the addresses to be updated before each replay without recapturing the graph. In our Inferact implementation, we use the same pattern. Operators using custom inference frameworks should verify their framework handles this correctly, as a misimplementation will produce incorrect outputs without raising an obvious error.

When CUDA graph replay is not worth enabling

We are not suggesting CUDA graph replay is universally worth enabling. There are clear cases where it adds complexity without sufficient benefit.

For prefill-heavy workloads where the ratio of prefill tokens to decode tokens is high (long-input, short-output tasks like summarization), the decode phase is a small fraction of total inference time. The per-token decode speedup affects a small portion of total wall-clock time, and the VRAM cost for graph objects may not be justified.

For small models on hardware where kernel launch overhead is already low relative to memory bandwidth constraints, the latency improvement from graph replay may be negligible. RTX 4090 PCIe systems, for example, are typically bandwidth-bound rather than kernel-launch-overhead-bound for decode, and the CUDA graph benefit is smaller than on NVLink-connected A100 systems.

The right approach: profile your decode step latency with and without graph replay on your actual hardware and model. The answer is in the trace, not in theoretical analysis.