Running INT4 quantized models alongside FP16 models on the same fleet is not a hypothetical scenario. It happens when you have hardware with different VRAM capacities and need to fit models that do not all land cleanly in FP16, when you want to maximize throughput on memory-constrained nodes without losing FP16 fidelity for requests that require it, or when you are evaluating whether the quality degradation from INT4 quantization is acceptable for a specific use case. The fleet ends up heterogeneous not just in hardware but in quantization format, and the scheduler needs to reason about both dimensions simultaneously.
This post covers how we handle quantization format routing in Inferact's scheduler, the quality signal we use to decide whether a request should go to an INT4 node or an FP16 node, and the edge cases that caused us the most trouble during implementation.
The Format Mismatch Problem
INT4 quantization (using approaches like GPTQ or AWQ) compresses the model weights from 16 bits to 4 bits per parameter, roughly halving memory requirements compared to BF16 and reducing memory bandwidth consumption during decode by approximately the same factor. The tradeoff is numerical precision: INT4 weights are dequantized at runtime, and the dequantization introduces small errors relative to the true FP16 weights. For most tasks on well-quantized models, the quality impact is small. For some tasks, it is not.
The scheduling problem is that the quality impact varies by request type. Mathematical reasoning tasks, code generation, and long chains of logical steps tend to be more sensitive to quantization noise than straightforward text completion or summarization. A scheduler that routes all requests uniformly to either format is either accepting unnecessary quality degradation for sensitive requests (if it routes everything to INT4) or leaving throughput on the table by never using INT4 at all.
Classifying Request Sensitivity to Quantization
The first challenge is building a fast classifier for quantization sensitivity that runs at scheduling time without adding significant latency. We do not want to run a separate model to classify the request; that would add latency overhead and require additional GPU resources. Instead, we use a rule-based classifier that inspects request properties at scheduling time.
The classification rules we have found most reliable in internal testing:
Requests with a temperature of 0 (greedy decoding) and a system prompt or task description that includes words associated with structured output (code, JSON, math, step-by-step reasoning) get flagged as quantization-sensitive. The rationale: at temperature 0, the model is being asked to reproduce a specific deterministic output, and quantization noise in the logits can shift which token wins the greedy selection at critical decision points.
Requests with temperature above 0.7 and task descriptions associated with creative or conversational output are classified as quantization-tolerant. At high temperature, sampling noise already dominates, and the marginal impact of quantization noise is small relative to the variation introduced by sampling.
Requests with very long output targets (estimated output tokens above 1000) receive a higher sensitivity score regardless of temperature, because quantization errors can compound over long sequences in ways that are hard to predict.
We want to be explicit that this classification is heuristic. We are not claiming that these rules reliably identify every request that will or will not notice quality degradation from INT4. They are a first-pass filter that routes the most obviously sensitive requests away from INT4 nodes while allowing quantization-tolerant requests to use the higher-throughput INT4 hardware. The right approach for any production deployment is to empirically evaluate the quality gap on your specific workload before relying on this routing.
The Throughput Impact of Precision-Tiered Routing
The throughput benefit of routing some requests to INT4 nodes is real but depends heavily on the fraction of requests classified as quantization-tolerant. If 80 percent of your requests fall into the tolerant category and 80 percent of those are successfully routed to INT4 nodes, you are effectively increasing your high-throughput capacity substantially for the bulk of your traffic.
In our internal benchmarks on a fleet with 4 FP16 nodes and 4 INT4 nodes serving Llama-3 13B workloads, precision-tiered routing on a workload where approximately 60 percent of requests were classified as quantization-tolerant produced roughly 35 percent higher aggregate tokens-per-second compared to routing all requests to FP16 nodes only. The INT4 nodes ran at significantly higher throughput because weight reads consumed less bandwidth, and the FP16 nodes were freed to handle the quality-sensitive requests without queuing behind tolerant requests.
Session Affinity and Format Consistency
One problem that is easy to miss in a stateless analysis: multi-turn conversations. A session that starts on an FP16 node will produce outputs consistent with FP16 precision throughout. If the same session is routed to an INT4 node mid-conversation due to load-balancing decisions, the model's effective behavior changes. For most users this is imperceptible. For users who are feeding the model's previous outputs back as context and relying on exact token-level consistency (for example, automated pipelines that check for specific output patterns), a precision switch mid-session can cause unexpected failures.
We implement session affinity by format: the first request in a session establishes the format tier (INT4 or FP16), and all subsequent requests with the same session ID are pinned to the same format tier. This does not mean they go to the exact same node; it means they go to a node in the same format class. The format tier is stored in the session metadata and checked at routing time.
The session affinity mechanism introduces a secondary scheduling constraint. When FP16 nodes are all busy and a new multi-turn request arrives for a session previously served by FP16, the request must wait for an FP16 slot even though INT4 slots are available. We accept this because the alternative (silently changing format mid-session) is worse. The practical impact on throughput is small in workloads where session continuity is concentrated on a minority of requests.
INT4 Kernel Performance on Different Hardware
The throughput advantage of INT4 quantization is not uniform across hardware. On A100 SXM4, the INT4 matrix-vector multiplication kernels (provided by GPTQ and AWQ inference libraries, or bitsandbytes) are well-optimized for the Ampere architecture and we see decode throughput improvements of roughly 1.7-1.9x compared to FP16 at the same batch size. On RTX 4090, the Ada Lovelace architecture has different INT4 matrix multiplication characteristics and the improvement is smaller, roughly 1.3-1.5x in our testing, depending on the specific quantization approach.
This means the throughput argument for routing requests to INT4 nodes is stronger on A100 than on RTX 4090. On an A100-only fleet, the case for precision-tiered routing is clearer because the throughput differential is larger. On a mixed A100/RTX fleet, the scheduler should also account for the hardware class of the INT4 nodes when estimating the throughput gain from routing there.
Format Metadata and Node Registration
A prerequisite for any of this to work is that the scheduler has accurate, up-to-date information about which quantization format is loaded on each node. We handle this through explicit node registration: when a node starts, it registers with the scheduler and includes its loaded model IDs, quantization formats, and VRAM capacity. The scheduler maintains a node state table that includes this metadata alongside live utilization metrics.
The metadata must be treated as immutable per node session. A node that re-registers with a different quantization format (for example, after a model reload) must be treated as a new node for routing purposes; any in-flight sessions that were pinned to the old format must not be routed to this node until they complete. We enforce this with a node quiescence period of 60 seconds after any re-registration event, during which the node accepts only new sessions rather than continuations of existing sessions from the old format.
What This Does Not Solve
Precision-tiered routing does not address the case where your FP16 nodes are consistently overloaded because the fraction of quantization-sensitive requests is too high. If more than 70 percent of your traffic requires FP16 precision, the INT4 nodes become underutilized and you have a fleet composition problem rather than a scheduling problem. In that case, re-evaluate whether your quantization sensitivity classification is too aggressive or whether your FP16 node count needs to increase.
The routing heuristics described here also do not provide quality guarantees. We can route quality-sensitive requests to FP16 nodes, but we cannot guarantee that the FP16 model will produce outputs of any particular quality. The quality floor is set by the base model and the serving configuration, not by the routing layer. The scheduler's role is to ensure that requests go to the highest-fidelity available format given the routing rules and the hardware state, not to validate the outputs.