A100 and H100 in the Same Fleet: Scheduling Without Waste

Hardware refresh cycles rarely align with when you actually need more capacity. For most early-stage teams running their own inference infrastructure, the practical reality is a fleet that accumulates incrementally: some A100 nodes acquired first, H100 nodes added later as they became available, possibly some RTX nodes for burst capacity. Running all of them with a simple round-robin load balancer leaves efficiency on the table in ways that are not obvious until you profile carefully.

This post describes the specific ways A100 and H100 differ in their inference performance characteristics, and how a hardware-aware scheduler can exploit those differences rather than averaging over them.

Why A100 and H100 are not just speed-binned versions of each other

It is tempting to think of H100 as "A100 but faster" and scale your routing proportionally. The H100 has roughly 2x the peak FLOPS of A100 for BF16 tensor core operations (989 TFLOPS vs 312 TFLOPS in their SXM variants). But peak FLOPS does not linearly translate to inference throughput for all workload types.

The more important differences for inference are in memory subsystem characteristics. H100 SXM5 has 3.35 TB/s HBM3 bandwidth versus A100 SXM4's 2.0 TB/s HBM2e. For memory-bandwidth-bound workloads (decode phase with small batch sizes, long-context attention over cached KV entries), the H100 advantage is larger than the compute ratio suggests. For compute-bound workloads (prefill with large batch sizes, large FFN layers with many tokens), A100 can keep pace with H100 at a fraction of the per-unit cost.

Second difference: H100 supports FP8 matrix multiplication natively, which A100 does not. An FP8-optimized inference stack on H100 can run 70B-parameter models at roughly 2x the token throughput of BF16, with acceptable quality degradation for most applications. If your A100 nodes are running INT4 or GPTQ quantization and your H100 nodes are running FP8, they have different effective capacity per unit of physical memory.

Third difference: NVLink bandwidth. H100 SXM NVLink 4.0 provides 900 GB/s of GPU-to-GPU bandwidth within a node, versus A100 SXM NVLink 3.0's 600 GB/s. For tensor-parallel inference where all-reduce operations synchronize between GPUs after each layer, this 50% bandwidth increase translates directly into lower communication overhead at larger tensor-parallel degrees.

What round-robin does wrong

Consider a fleet of 4 A100 nodes and 2 H100 nodes serving a mixed workload: short-input/long-output requests (latency-sensitive, benefits from decode-speed advantage of H100) and long-input/short-output requests (compute-bound prefill, A100 is adequate).

Round-robin assigns equal shares of each request type to all nodes. The H100 nodes process short-input/long-output requests faster than the A100 nodes, so H100 queue depth drops while A100 queue depth grows. A basic weighted round-robin by capacity ratio helps, but it still sends equal ratios of each request type to both hardware generations. The H100's decode advantage is not concentrated on the requests that benefit from it most.

Under bursty traffic, the A100 nodes saturate first because they process each request type more slowly. The H100 nodes still have headroom but cannot absorb the excess because the load balancer has already assigned those requests to A100 queues. Effective fleet utilization drops to the utilization ceiling of the A100 nodes.

Hardware-aware routing in the Inferact scheduler

The Inferact scheduler maintains a node capability registry that stores, for each node, the GPU architecture, HBM bandwidth, KV cache capacity in bytes, and whether FP8 or INT4 quantization is loaded. This registry is populated at node registration and updated on configuration changes.

Routing decisions use two request-level signals that map to different hardware strengths. The first is the input-to-expected-output token ratio. For requests with high input length and low expected output (classification, summarization, structured extraction), the work is prefill-dominant. For requests with low input and high expected output (code generation, chat completion with long responses), the work is decode-dominant. The expected output length can come directly from the API request if provided, or from a lightweight output-length estimator trained on recent request history.

Decode-dominant requests route preferentially to H100 nodes. Prefill-dominant requests route to whichever node has the highest available prefill bandwidth, which may be A100 if the H100 nodes are currently carrying high decode-dominant load. The scheduler does not rigidly segregate by hardware type: if H100 nodes are underloaded, prefill-dominant requests go there too. The preference is soft, expressed as a routing weight rather than a hard filter.

KV cache capacity asymmetry

Both A100 and H100 SXM flagship variants carry 80GB HBM, so raw VRAM is the same. But the effective KV cache capacity after model weights are loaded differs depending on what is running. An 8xA100 node running Llama-3 70B BF16 (140GB for weights, across 8 GPUs at 17.5GB each) leaves roughly 62.5GB per GPU for KV cache. An 8xH100 node running the same model in FP8 (70GB for weights, 8.75GB each) leaves roughly 71.25GB per GPU for KV cache. This is a 14% larger KV cache on H100 nodes even with identical VRAM capacity.

The Inferact scheduler accounts for this by tracking available KV cache pages per node, not just queue depth. A request with a very long context that would require many KV cache pages routes away from nodes where those pages would cause significant eviction of other active sequences. This cross-node KV cache visibility prevents the scenario where one node's cache is thrashing while another has ample headroom.

Tensor parallelism degree matching

Tensor parallelism (TP) splits the model across multiple GPUs, requiring all-reduce communication between GPUs after each transformer layer. The optimal TP degree depends on the all-reduce communication overhead relative to computation time. On A100 with NVLink 3.0 (600 GB/s), TP-8 (splitting across all 8 GPUs in a node) adds meaningful communication overhead for models where FFN layers are compute-light. On H100 with NVLink 4.0 (900 GB/s), TP-8 can be used more freely because the all-reduce time is proportionally lower.

Some teams running mixed A100/H100 fleets configure the same TP degree on both node types for operational simplicity. This works but leaves H100 nodes running a sub-optimal TP configuration. The Inferact scheduler supports per-node TP configuration and can route requests to nodes based on whether the model configuration on that node matches the request's latency requirements. A request with a strict p50 TTFT SLO might route preferentially to a TP-8 H100 node rather than a TP-4 H100 or A100 node, even if other nodes have lower queue depth.

Practical steps for heterogeneous fleet operators

Profile your actual workload distribution before designing routing rules. The inputs-to-outputs ratio for your production traffic may not match your intuitions. We have seen teams assume their workload is decode-heavy and build routing for that, only to find that their p95 request has a 4:1 input-to-output ratio. Measuring before designing is worth two days.

Start with soft routing preferences rather than hard filters. Hard filters that strictly restrict request types to specific node generations create availability cliffs: if all your H100 nodes are occupied and a latency-sensitive request arrives, it queues rather than routing to an available A100. A soft preference allows fallback while keeping the routing bias in the right direction under normal load.

Monitor per-node KV cache utilization separately from queue depth. These metrics diverge under long-context loads. High queue depth on A100 nodes combined with high KV cache utilization on H100 nodes (despite lower queue depth) indicates you are not fully exploiting H100's larger effective KV cache. Adjusting the routing weight to send more long-context traffic to H100 nodes may improve overall fleet efficiency.

The goal is not to pretend the hardware is uniform. The goal is to route each request to the node that will process it most efficiently given the node's specific compute and memory profile. That requires knowing what you have and what each request needs.