Interconnect: memory at a longer distance

Because every technique that works on the memory hierarchy transfers, and treating them separately means rediscovering each one twice with different vocabulary.

Run the correspondence explicitly:

Every row is the same idea at a different constant. Gradient bucketing in DDP is coalescing. Overlapping the backward pass with the gradient all-reduce is double buffering. ZeRO’s choice to shard optimiser state and all-gather parameters on demand is a capacity-versus-traffic tiling decision, decided by exactly the reasoning of Module 7. FSDP’s reshard_after_forward flag is literally a keep-resident-or-refetch choice.

The framing also produces a genuinely useful design number. Ask: what is the arithmetic intensity my network requires? An H100 at ~1 PFLOP/s BF16 with 900 GB/s of NVLink needs roughly 1100 FLOP per byte crossing NVLink to stay compute-bound, and about 20,000 FLOP per byte crossing a 50 GB/s NIC. Those two numbers explain the entire parallelism strategy landscape at a glance: tensor parallelism communicates activations every layer and therefore needs the 1100-class link, while data parallelism communicates gradients once per step and comfortably tolerates the 20,000-class one. That is not a heuristic anybody had to invent — it falls straight out of roofline applied one level down the ladder.

Where the analogy genuinely breaks, and it is worth being precise rather than pretending it does not: memory is passive and point-to-point, while a network has contention between independent endpoints and topology-dependent effective bandwidth. Two GPUs can each be doing something sensible and collide in a shared switch, and that failure has no memory-hierarchy analogue. It is also why a collective’s real cost depends on how the algorithm maps onto the physical topology, which is Module 11.

It exists because of a discontinuity in the ladder at the package and chassis boundary, and it is being attacked from both sides because that discontinuity is expensive.

Inside a node, links are short, electrical, and can run wide and fast: NVLink 4 gives 900 GB/s per GPU and NVLink 5 gives 1.8 TB/s, with an NVSwitch fabric providing full all-to-all inside the domain. Leave the chassis and you are on a NIC, through a switch, over a cable, with error handling, a protocol stack and roughly 20× less bandwidth at 3–5× the latency. That is not a gradual decline, it is a cliff, and it is a cliff for physical reasons: the number of SerDes you can put on a die perimeter, the power to drive a signal a metre versus a centimetre, and the cost of the optics or cabling.

Everything about how large models are parallelised is a response to this cliff. The rule that has emerged, and it is worth stating as a rule:

Put the communication-intensive parallelism inside the high-bandwidth domain, and the communication-light parallelism across it.

Which in practice means tensor and expert parallelism within an NVLink domain or a TPU pod slice, pipeline parallelism across the boundary (point-to-point, small, and overlappable), and data parallelism outermost (one all-reduce per step, latency-tolerant, bucketable). Once you have the FLOP-per-byte numbers above, this is not a rule of thumb — it is the only assignment that fits.

Where it is going, on both sides:

• The scale-up domain keeps growing. 8 GPUs, then NVL72’s 72, with roadmaps beyond. Every GPU moved inside the domain is a set of transfers that no longer crosses the cliff. This is the single most consequential system-architecture trend of the last few years, and it is why rack-scale is suddenly the unit of design.
• The cliff is being lowered. Co-packaged optics moves the electrical-to-optical conversion onto the package, cutting the power and the distance penalty. UCIe standardises die-to-die at sub-HBM energy per byte. Ultra Ethernet is bringing scale-out closer to scale-up semantics.
• And a note on TPUs, which took a different route entirely: no switched fabric, a 2D or 3D torus with nearest-neighbour links only. That is much cheaper — no switches at all — and it works because the collectives that matter map beautifully onto a torus, at the cost of arbitrary point-to-point patterns being genuinely bad. A topology chosen for a known set of access patterns, which is precisely the scratchpad-versus-cache argument from Module 3 applied to the network.

Because bandwidth optimality is one of three terms, and the other two are what actually bite in practice.

The cost model for a collective is roughly

$T = \alpha \cdot (\text{hops}) + \beta \cdot (\text{bytes}) + \gamma \cdot (\text{reduction work})$

with $\alpha$ the per-hop latency, $\beta$ the inverse bandwidth. Ring all-reduce minimises the $\beta$ term — $2S(N-1)/N$ bytes per endpoint, asymptotically $2S$ and independent of $N$, which is provably optimal. But it takes $2(N-1)$ steps, so the $\alpha$ term grows linearly in $N$. At $N = 512$ with a 2 µs per-hop latency, that is about 2 ms of pure latency before any bytes are considered. For a large gradient buffer that is negligible; for a small one it is the entire cost.

Hence the real behaviour:

• Small $S$: use a tree, or a latency-optimised algorithm. $O(\log N)$ steps instead of $O(N)$, and the extra bandwidth is irrelevant because there was not much data. NCCL and similar libraries switch algorithm by message size at runtime, which is why their tuning tables exist.
• Large $S$: use a ring, or a hierarchical composition. Bandwidth dominates and ring is optimal.
• In between, and in practice: hierarchical. Reduce-scatter within the node over NVLink, all-reduce across nodes over the slow network with $1/G$ of the data (where $G$ is GPUs per node), then all-gather within the node. This is the standard construction and it is a direct application of the memory-hierarchy principle: do as much as possible at the cheap level so the expensive level sees less traffic.

Three further complications that the clean formula hides:

• Topology mapping. A “ring” is a logical construct; its performance depends on whether consecutive logical ranks are physically adjacent. A ring embedded badly in a fat tree traverses the same congested switch repeatedly. This is why rank-to-node assignment measurably changes training throughput, and why libraries invest so much in topology detection.
• Overlap changes the objective. If the collective overlaps with computation, its latency stops mattering and only its bandwidth consumption does, since it is now competing with nothing but itself. That flips the algorithm choice, and it is why bucketing granularity in DDP is a tuned parameter: too-small buckets are latency-dominated, too-large ones start after the compute they should have overlapped with has finished.
• In-network reduction. Switches that can perform the reduction (SHARP and similar) change the formula rather than the constants, removing an entire traversal. When available it is a large win, and it is the network equivalent of processing-in-memory — with the advantage that here it actually shipped.

It is the observation that off-package bandwidth is bounded by the perimeter of the die and the package while compute is bounded by their area — and area grows faster than perimeter. That is a geometric constraint, not an engineering one, and it means the FLOPs-to-off-chip-bytes ratio degrades every generation no matter how well anyone executes.

The consequences run through the entire series, and it is worth seeing them as consequences of one thing:

• The rising ridge point (Module 2). Each generation needs more arithmetic per byte to reach peak, so the tile sizes that saturate the machine grow, and mappings that were fine become bandwidth-starved without anyone changing the code.
• The push to lower precision (Module 10). Halving bytes is the most direct way to buy back effective bandwidth on a channel whose width you cannot increase. It is not fundamentally about arithmetic speed — that is the smaller half of the benefit.
• The growth of on-die SRAM (Module 3). If bytes cannot cross the boundary quickly, keep more data on the correct side of it. Taken to its limit you get Cerebras: 44 GB of on-wafer SRAM at ~21 PB/s and no HBM at all, which is the beachfront limit answered by refusing to have a beachfront.
• Chiplets and advanced packaging. UCIe at ~0.5 pJ/byte is cheaper than HBM, so aggregating several dies into one logical package extends the “inside” region. The package becomes the unit of design precisely because the boundary is what is expensive.
• Growing scale-up domains. NVL72 exists so that more communication happens inside the high-bandwidth region. Every endpoint pulled inside is traffic that never pays the cliff.
• Co-packaged optics. Attacks the constraint most directly by improving bandwidth density and energy per bit at the boundary itself — the only intervention that raises the ceiling rather than avoiding it.

And the software consequence, which is this series’ actual subject: as the ratio worsens, an increasing fraction of achievable performance is determined by the mapping rather than by the hardware. A chip with twice the FLOPs and 1.4× the bandwidth is not 2× faster on anything; it is 2× faster only on mappings good enough to exploit it, and those mappings did not exist when the chip was designed. That gap — between what the silicon can do and what the compiler can reach — is what Part R is about, and it widens every generation for reasons no hardware team can fix.