The RTX 4090 is not designed for data center inference. It has no NVLink, its 24 GB GDDR6X pool is less than a third of what an A100 SXM4 provides, and the PCIe 4.0 x16 interface limits host-to-device transfers to about 32 GB/s in each direction. But it is available, the price-per-FLOP ratio is competitive, and teams building proof-of-concept inference pipelines or running small production workloads on constrained budgets often land on it. Configured poorly, it delivers mediocre performance. Configured well, it can sustain reasonable single-GPU throughput for Llama-3 8B workloads.
These notes cover the specific PagedAttention configuration choices that matter on an RTX 4090. We run a small cluster of these cards as part of our internal test environment for Inferact, and the settings here reflect what produced the best observed throughput in our testing across several workload types.
The Constraint That Changes Everything: VRAM Is Not Abundant
Llama-3 8B in FP16 requires approximately 16 GB for model weights. An RTX 4090 has 24 GB GDDR6X. That leaves roughly 8 GB for KV cache pages after accounting for weights and a small runtime overhead (CUDA context, activations during forward pass). Depending on your block size and context length targets, 8 GB is tight.
At the default vLLM block size of 16 tokens, each KV cache block for Llama-3 8B in FP16 occupies approximately 0.5 MB (32 layers, 2 heads per layer for GQA, 128 head dimension, 16 tokens, 2 bytes per element for FP16). With 8 GB available for KV cache, that is approximately 16,000 blocks, supporting a theoretical maximum concurrent token budget of 256,000 tokens. In practice, the usable capacity is lower because some blocks are reserved for in-flight prefill phases and allocation overhead.
The practical limit we have observed in internal testing: an RTX 4090 node can comfortably serve approximately 40-60 concurrent 2K-context sequences at moderate throughput targets. Beyond that, KV cache pressure starts producing evictions that degrade TTFT.
Block Size Selection
The block size parameter in PagedAttention implementations controls the granularity of KV cache allocation. Smaller blocks reduce internal fragmentation (a sequence that generates 17 tokens wastes only 15 tokens worth of space in a 32-token block, versus 31 tokens in a block that holds 32 tokens). Larger blocks improve memory access locality during decode because more tokens are physically contiguous.
On the RTX 4090, we found that a block size of 16 tokens (vLLM's default) is generally optimal for the workloads we test. Here is the reasoning: the GDDR6X memory system performs well on coalesced accesses but does not benefit dramatically from very large contiguous allocations the way HBM does. The RTX 4090's memory controller is optimized for gaming workloads, which involve frequent small random accesses to texture memory. The result is that the locality benefit of larger blocks is less pronounced than on A100.
We did test block sizes of 32 and 64 tokens. At block size 32, we saw roughly 3 percent higher throughput on long-context sequences (8K+) due to reduced page table lookups, but we also saw higher tail latency variance because large blocks amplify the cost of a single eviction event. At block size 64, fragmentation became a problem on the constrained 8 GB KV budget. Block size 16 remains our recommendation for this hardware.
The GPU Memory Utilization Parameter
vLLM exposes a gpu_memory_utilization parameter (default 0.90) that controls what fraction of total GPU memory is reserved for the KV cache and model weights combined. On a node with abundant VRAM like an A100 80GB, leaving 10 percent free provides ample safety margin. On an RTX 4090 with 24 GB, the remaining 10 percent is only 2.4 GB.
We typically set gpu_memory_utilization=0.92 on RTX 4090 nodes. This recovers an additional 480 MB for KV pages (roughly 960 additional blocks at block size 16) while keeping enough headroom to avoid CUDA out-of-memory conditions during prefill spikes. We do not recommend going above 0.93 without monitoring memory pressure carefully. The risk is not a hard OOM during normal operation but rather an OOM during a large batch prefill when activation memory temporarily exceeds the residual free space.
One important note: the gpu_memory_utilization parameter does not account for the CUDA runtime context overhead, which on Ada Lovelace cards (RTX 4090) is approximately 400-600 MB depending on the driver version. Factor this into your actual available KV cache budget when estimating capacity limits.
PCIe Bandwidth as the Real Ceiling
On server-class hardware like the A100, model weights reside in GPU memory and the bottleneck during decode is HBM bandwidth reading those weights per forward pass. The same is true on RTX 4090, but there is an additional constraint that matters during initial model loading and during any scenario involving weight swapping: PCIe 4.0 x16 limits host-to-device transfers to roughly 32 GB/s.
Loading a 16 GB Llama-3 8B FP16 model from host RAM over PCIe 4.0 takes approximately 500 ms in the optimal case. If you have a fleet management layer that swaps models between requests (for example, switching between Llama-3 8B and a 13B model depending on routing rules), this transfer latency becomes a hard floor on your cold-start penalty. On A100 NVLink systems, inter-GPU weight transfers happen at 600 GB/s or better, making this a non-issue. On RTX 4090, it is a real constraint.
Our recommendation: on RTX 4090 nodes, commit to a single model per node and do not swap weights during operation. The throughput cost of mid-session weight swaps over PCIe is prohibitive. If you need multiple models on the same physical machine, run multiple vLLM instances on separate processes, each bound to a different GPU (if you have multiple cards per machine).
Tensor Parallelism on RTX 4090
We are often asked whether running tensor parallel inference across two RTX 4090 cards makes sense. The short answer is generally no, at least not for Llama-3 8B. The reason is the communication path. Two RTX 4090 cards in a desktop or workstation chassis communicate over PCIe, not NVLink. An all-reduce across two cards over PCIe during each transformer layer adds approximately 2-4 ms per forward pass in our measurements, depending on activation size and system bus utilization. For Llama-3 8B, each single-GPU forward pass takes on the order of 20-30 ms at batch size 1. Adding 2-4 ms per forward pass in communication overhead for a 2-GPU TP setup degrades single-request latency by 10-15 percent while only doubling the KV cache budget.
If your primary motivation for TP on RTX 4090 is expanding the KV cache pool, consider using pipeline parallelism at the model-serving layer instead: run two independent single-GPU inference processes and have your scheduler distribute requests across them. You get the same aggregate KV capacity without paying the all-reduce overhead on every forward pass.
Practical Flags for vLLM on RTX 4090
Based on our internal testing, the flags that matter most when launching vLLM on an RTX 4090 serving Llama-3 8B:
--block-size 16 as discussed above. --gpu-memory-utilization 0.92 to recover additional KV pages. --max-num-seqs 64 to cap concurrent sequences at a level where KV cache pressure remains manageable. --max-model-len 8192 if your workload does not need longer contexts; this prevents long-context requests from consuming a disproportionate share of the KV budget. --enforce-eager is sometimes recommended for Ada Lovelace but we have found it unnecessary for inference-only workloads and it disables CUDA graph capture, which costs a few percent of throughput. Leave it off unless you are seeing CUDA graph compilation failures.
The --max-num-seqs limit is not something vLLM will automatically tune for you based on hardware. It defaults to 256, which is too high for a 24 GB card serving FP16 models at moderate context lengths. We set it based on the formula: available KV blocks divided by the average context length divided by the block size, with a safety factor of 0.7. For the RTX 4090 with our workload mix, this works out to about 60-70 concurrent sequences, so we round down to 64.
What We Are Not Claiming
These notes are not an argument that RTX 4090 is a good choice for production inference at scale. It is not. A single A100 80GB node outperforms a single RTX 4090 by approximately 2x on decode throughput and allows far larger KV cache budgets for long-context workloads. The RTX 4090 has a place in development environments and small-scale production deployments where cost per GPU is the binding constraint, not throughput per dollar at scale.
The configuration choices here also reflect our specific workload mix and our specific vLLM version. KV page sizing calculations depend on the model's GQA configuration, which differs across model families. Before adopting these numbers for a different model, recalculate the per-block memory footprint using the actual number of KV heads, head dimension, and block size for that model.