Most GPU fleets that run open-weight models in production are not homogeneous. A team starts with a few RTX 4090 nodes for initial testing, then acquires A100 SXM nodes as load grows. The result is a fleet where two classes of hardware coexist: the A100s with their 80 GB HBM2e and 2 TB/s memory bandwidth, and the RTX 4090s with 24 GB GDDR6X at roughly 1 TB/s. Running these in a single pool without a hardware-aware scheduler wastes the A100s, overloads the RTX nodes, or both simultaneously.
This post documents what we have learned building Inferact's heterogeneous batch scheduler. Most of the hard lessons came during a deployment we ran in early 2026 for an internal test fleet combining four A100 SXM4 nodes and eight RTX 4090 nodes serving Llama-3 8B and 13B workloads.
Why Naive Round-Robin Fails
The intuitive starting point is distributing requests in round-robin across all available nodes. This fails for a predictable reason: weight loading and KV cache pressure interact differently on different hardware.
On an RTX 4090 serving Llama-3 8B in FP16, the 16 GB model weights consume roughly 67 percent of the 24 GB VRAM, leaving about 7 GB for KV cache pages. An A100 80GB node serving the same model in FP16 has approximately 63 GB remaining for KV cache after weights. That is a 9x difference in available KV cache budget. Round-robin distributes sequences without regard to this asymmetry. When a batch of long-context requests arrives, the RTX nodes exhaust their KV page pool while A100 nodes still have substantial headroom. The RTX nodes begin evicting KV pages, which forces cache refills on subsequent decode steps, degrading tail latency significantly.
In our internal benchmarks on the mixed fleet, a naive round-robin scheduler produced p99 time-to-first-token (TTFT) values roughly 3.4x higher than a hardware-aware scheduler on 4K-context workloads, and the gap widened further at 16K context lengths. The mean TTFT was only 1.2x worse, which is why looking only at average latency hides the problem.
Classifying Workloads for Hardware Assignment
The scheduler needs to classify each incoming batch along two axes before assignment: expected sequence length and quantization format on the target node.
For sequence length, we bucket incoming requests at scheduling time rather than predicting exact output lengths. Requests carrying a system prompt over 2K tokens almost certainly belong on nodes with larger KV cache budgets. We use a simple threshold: requests where (input_tokens + estimated_output_tokens) exceeds 4096 get routed preferentially to A100 nodes. Below that threshold, either hardware class is acceptable and load governs the decision.
The quantization dimension is less obvious but equally important. When nodes load different quantization formats of the same model (say, one node runs INT4 GPTQ and another runs BF16), the scheduler must track which format is resident on which node and assign requests accordingly. We do not recommend mixing quantization formats within a single inference request batch. The arithmetic throughput difference between INT4 and BF16 creates timing skew in batched decode steps, which complicates synchronization if you ever need cross-node tensor parallelism. Keep each node's quantization format stable within a session and route based on per-node format metadata.
Memory Bandwidth Estimation at Schedule Time
A useful signal the scheduler can compute cheaply is instantaneous memory bandwidth utilization per node. During autoregressive decode, inference is memory-bandwidth-bound, not compute-bound. The number of tokens you generate per second is approximately proportional to your available bandwidth divided by the bytes that must be read per forward pass.
For a node running Llama-3 8B in BF16, each decode step reads roughly 16 GB of weights. On an RTX 4090 with 1 TB/s bandwidth, the theoretical maximum decode throughput is about 62 tokens per second for a batch of one. In practice, with the overhead of KV cache reads, CUDA kernel launch, and GPU scheduling, our internal measurements show about 47 tokens per second for a single-request batch on a fresh node. On A100 SXM4, the same configuration runs at approximately 105 tokens per second single-request, reflecting the 2x bandwidth advantage.
When you track actual bandwidth utilization in real time, you can predict whether a new batch will push a node into its bandwidth saturation zone. A node already at 85 percent of bandwidth utilization will add only marginal throughput for the next batch while contributing to latency for existing in-flight sequences. The hardware-aware scheduler uses this signal to bias routing away from saturated nodes even when their queue depth appears modest.
Weight Assignment Across Node Classes
When a fleet is large enough to pin different models to different node classes, weight assignment becomes a scheduling problem in its own right. The general principle is: assign larger models to nodes with the highest memory capacity, and reserve smaller models for hardware where they fit comfortably with room for generous KV cache.
For our internal test fleet, we settled on this assignment: Llama-3 70B (BF16, 140 GB) runs exclusively on multi-GPU A100 pairs using tensor parallelism across two cards. Llama-3 13B (INT4 GPTQ, approximately 7 GB) can run on either node class, but we pin it to A100 nodes by default because the INT4 kernel performance is better on Ampere than on Ada Lovelace at the batch sizes we see. Llama-3 8B (BF16) is the only model we assign to RTX 4090 nodes directly.
The key lesson here: do not determine weight assignment purely from model file size. Factor in the KV cache budget you will need given your expected context lengths. A model that fits in VRAM with 8 GB to spare is fine for 2K-context workloads but may be unusable for 16K-context workloads if the KV cache would require more than that 8 GB headroom.
What Breaks When Quantization Formats Differ Across Nodes
The most common failure mode in mixed-format fleets is output quality inconsistency that is difficult to attribute to the infrastructure layer. A user submitting requests that happen to land on INT4 nodes may notice subtly different output distributions compared to requests landing on BF16 nodes. This is not a bug in the scheduler but it creates a confusing debugging experience.
We handle this in two ways. First, for any request that includes a session ID (multi-turn conversation), we implement session affinity: the request always routes to the same node class that handled the first turn. This ensures that within a conversation, the user sees a consistent model variant. Second, we expose a per-request routing hint in our API that allows callers to specify a precision preference. Most callers do not use this, but it has proved valuable for teams running evaluation pipelines where they need exact reproducibility.
A more subtle breakage occurs when you add a new node with a different CUDA version or driver stack. Inference kernel behavior can differ across driver versions in ways that affect output determinism. We track the exact driver and CUDA toolkit version per node and flag mismatches during fleet health checks. This is not a scheduler issue per se, but a healthy heterogeneous scheduler needs access to this metadata to reason correctly about node equivalence.
Starvation Prevention
In a priority-based heterogeneous scheduler, low-priority requests can starve indefinitely if high-priority long-context requests continuously occupy A100 nodes. We prevent this with an aging mechanism: requests that have waited beyond a configurable threshold (we default to 2x the median TTFT observed in the last 60 seconds) get their priority boosted to match the current top-priority class. This means that, in the worst case, a batch of short requests waiting behind a queue of 32K-context requests will eventually get promoted and cleared, even if the A100 nodes remain busy.
The aging interval needs calibration per workload. Set it too low and you effectively collapse priority tiers. Set it too high and tail latency for short requests under heavy long-context load remains unacceptably high. We expose this as a configurable parameter because the right value depends heavily on the ratio of long-context to short-context traffic in your specific workload mix.
Where This Leaves Us
Heterogeneous scheduling is not a solved problem. The approaches described above work well for the workloads we have tested, but they make assumptions that may not hold for every fleet. Notably, we assume that weight loading is static per node, and we do not currently handle dynamic weight swapping. If your fleet needs to swap models on demand across different hardware classes, the scheduler needs to track warm-up latency as a cost that factors into routing decisions, which adds complexity we have not yet fully addressed.
We are also not claiming that the bandwidth utilization signal is sufficient on its own. At very high batch sizes, compute saturation starts to dominate on A100 nodes while RTX nodes remain bandwidth-bound. Handling that regime requires tracking both bandwidth and compute utilization simultaneously. That work is in progress.
The core principle stands: treat each node as having a specific resource profile rather than an interchangeable slot in a pool. The scheduler that wins is the one that accurately tracks those profiles and makes routing decisions that respect them.