Tensor parallelism splits a model's weight matrices across multiple GPUs so that each card holds and processes a shard of each layer, with all-reduce operations synchronizing partial results at the end of each layer. Inside a single server node with NVLink, this works well: an 8x A100 SXM4 node has 600 GB/s NVLink bandwidth between cards, and the all-reduce latency for typical activation sizes is small enough that TP=8 is practical for large models. Across nodes, the picture changes. The communication path shifts from NVLink to whatever network fabric connects the nodes, and the all-reduce overhead grows to the point where it consumes a significant fraction of each forward pass.
This post documents what we observed when profiling multi-node tensor parallelism on A100 setups where NVLink is the intra-node fabric but the inter-node connection is either 100 Gb/s InfiniBand or 25 Gb/s Ethernet. The results shaped how we handle TP degree selection in Inferact's fleet scheduler.
Why All-Reduce Overhead Matters for Inference
During a transformer forward pass with tensor parallelism, each attention and MLP block requires an all-reduce operation to sum partial activations across all TP ranks before passing the result to the next layer. For a 70B model with 80 layers, that is 160 all-reduce operations per forward pass (one after attention, one after MLP for each layer). Each all-reduce transmits a tensor of size (batch_size * sequence_length * hidden_dim) bytes, where hidden_dim is 8192 for Llama-3 70B.
For a batch of 16 sequences at 512 tokens each, the activation tensor per all-reduce is approximately 16 * 512 * 8192 * 2 bytes (BF16) = 134 MB. Across 160 layers, the total all-reduce data volume per forward pass is roughly 21 GB. At NVLink's 600 GB/s effective bandwidth, this takes approximately 35 ms. At 100 Gb/s InfiniBand (about 12.5 GB/s), it takes approximately 1.7 seconds, which is completely impractical.
In practice, the all-reduce is pipelined with compute and the effective bandwidth utilization is not 100 percent, so the real numbers are somewhat better than this calculation suggests. But the fundamental constraint stands: cross-node all-reduce on commodity networking is not viable for tensor parallelism at 70B scale.
What We Measured
We profiled three configurations on our internal test cluster for Llama-3 70B inference:
Configuration 1: TP=8, single A100 SXM4 node (8 GPUs, NVLink only). This is the intra-node baseline. All-reduce occurs exclusively over NVLink at 600 GB/s. In our internal benchmarks, this configuration produced a median forward pass latency of approximately 38 ms at batch size 16, sequence length 512.
Configuration 2: TP=16, two A100 SXM4 nodes connected via 100 Gb/s InfiniBand HDR. Each node contributes 8 GPUs. Intra-node communication goes over NVLink; inter-node communication goes over IB. We used NCCL for all-reduce and measured with NCCL's ring topology. Median forward pass latency: approximately 110 ms at batch size 16, sequence length 512. That is 2.9x slower than TP=8 on a single node, and we are using 2x the hardware.
Configuration 3: TP=16, two A100 nodes connected via 25 Gb/s Ethernet. NCCL with Ethernet. We ran this primarily to characterize what happens on fleets that do not have InfiniBand. Median forward pass latency: approximately 380 ms at batch size 16, sequence length 512. This is effectively unusable for interactive inference workloads.
The IB configuration is interesting because it is technically viable for throughput-focused workloads but at a cost. If you are processing documents in batch mode and care only about total throughput, the 110 ms forward pass latency may be acceptable. If you are serving interactive requests with p99 TTFT targets below 2 seconds, it is not.
The NVLink Topology Detail That Surprised Us
One finding that caught us off guard: NVLink is not the full solution even within a single server node when the topology is non-uniform. Not all A100 DGX A100 configurations have full-mesh NVLink. Some multi-GPU server configurations use a NVLink switch that creates a two-hop topology between certain GPU pairs. In a 2-hop NVLink topology, the effective bandwidth between two GPUs that are not directly linked is roughly half of the direct-link bandwidth.
We observed this on a 4-GPU node in our test cluster that used NVLink but not with the full DGX topology. The all-reduce for a TP=4 configuration showed 30-40 percent higher latency than expected based on the nominal NVLink bandwidth. Profiling with NCCL's topology detection revealed the two-hop path. On a true DGX A100 with NVSwitch, every GPU-to-GPU path is a single hop and this is not an issue. On other multi-GPU server configurations, verify the actual NVLink topology before assuming you will get full NVLink bandwidth for all-reduce.
Practical Alternatives for Large Models
If cross-node tensor parallelism is impractical, what are the alternatives for serving a 70B model that does not fit on a single node?
Pipeline parallelism is the primary alternative. In pipeline parallelism, each node holds a contiguous range of layers rather than shards of all layers. Forward pass data flows sequentially from node 1 (layers 1-20) to node 2 (layers 21-40) to node 3 (layers 41-60) to node 4 (layers 61-80) for a 4-node pipeline. Inter-node communication in pipeline parallelism transmits only the activation tensor between adjacent pipeline stages, not the all-reduce volume. For Llama-3 70B with hidden_dim 8192, the activation tensor between stages is approximately 16 * 512 * 8192 * 2 bytes = 134 MB per batch at the settings above. At 100 Gb/s IB, this transfers in about 11 ms, a manageable overhead.
Pipeline parallelism introduces pipeline bubbles: periods where some pipeline stages are idle because the forward pass has not yet reached them or the backward pass has not yet returned (for training; inference pipeline bubbles are smaller but still exist). For inference, the bubble overhead is primarily visible as increased TTFT because the first forward pass through the pipeline must traverse all stages before the first output token is produced. The inter-token generation rate after the first token can be pipelined more efficiently.
In our internal testing, a 4-stage pipeline parallel configuration for Llama-3 70B on 4 A100 nodes connected via 100 Gb/s IB produced a median forward pass latency of approximately 52 ms at batch size 16, compared to 110 ms for TP=16 on the same hardware. The TTFT was higher for small batches due to pipeline startup latency, but the sustained decode throughput was significantly better.
Combining TP and PP
The approach we use for large models in production test setups is a combination: tensor parallelism within each node (TP=8 over NVLink) and pipeline parallelism across nodes (PP=2 for 70B on 2 nodes). This gives TP=8, PP=2, total 16 GPUs. Intra-node communication uses NVLink at full speed; inter-node communication is limited to the pipeline activation transfers, not the full all-reduce volume.
This hybrid approach is described in the Megatron-LM literature and is commonly used in training. For inference, it requires that the scheduler understand the pipeline topology and assign entire batches to a single pipeline rather than distributing individual requests across different pipeline stages. Our scheduler tracks which node group forms each pipeline and routes requests accordingly.
TP Degree Selection in the Scheduler
The fleet scheduler needs to track TP configuration per node group and factor this into routing decisions. A single-node TP=8 group and a 2-node TP=8 group (using cross-node NVLink, if available) have different latency profiles even though both involve 8 GPUs. Our node registration protocol requires each node group to declare its TP degree, PP degree, and the interconnect type (NVLink-only, NVLink+IB, Ethernet) so the scheduler can apply realistic latency estimates when making routing decisions.
For interactive workloads, the scheduler prefers single-node TP groups regardless of their smaller model capacity, because the latency variance of cross-node TP is harder to bound. For throughput-focused workloads where requests sit in queue anyway, the scheduler is willing to route to cross-node TP groups to maximize hardware utilization.
A Caveat on These Numbers
All latency numbers in this post come from our internal test cluster and specific NCCL and CUDA configurations. Real-world numbers will vary based on server chassis, NVLink topology, IB switch latency and fabric congestion, NCCL version, and the specific CUDA graph capture state of your inference engine. Before drawing architectural conclusions from these figures, profile your own hardware with a representative workload. The qualitative relationships (NVLink much better than IB, IB much better than Ethernet, PP better than TP for cross-node communication) are robust, but the quantitative multipliers are hardware-specific.