Parallelism is mapping onto the interconnect

Both are all-reduces. What differs is how many FLOPs of work each byte buys, and the ratio differs by three orders of magnitude.

Tensor parallelism. Split a transformer layer’s hidden dimension across $t$ devices. The two GEMMs of the MLP block do $O(BSH^2)$ FLOPs, and the column-then-row split means each device holds a partial sum over its shard of $H$, so the results must be all-reduced: $O(BSH)$ bytes, twice per layer (once for attention output, once for the MLP output). The ratio is

$\frac{\text{FLOPs}}{\text{bytes}} \approx \frac{BSH^2}{BSH} = H$

For $H = 8192$ that is 8192 FLOP per byte. Compare against the requirement: an H100 at ~1 PFLOP/s BF16 with 900 GB/s of NVLink needs 1100 FLOP/byte to stay compute-bound, and with a 50 GB/s NIC needs 20,000. So TP over NVLink has 7× headroom and TP over Ethernet is short by 2.4×, meaning the network becomes the bottleneck and more than half the machine idles. Note the ratio is $H$ and does not improve with batch size — both terms scale with $B$ — which is why you cannot batch your way out of a bad TP placement, and why TP degree is capped at the size of the high-bandwidth domain.

Data parallelism. Replicate the model, shard the batch. Gradients are all-reduced once per step: $2P$ bytes for $P$ parameters. The step performs roughly $6 P N_{\text{tok}}$ FLOPs for $N_{\text{tok}}$ tokens. The ratio is

$\frac{6PN_{\text{tok}}}{2P} = 3N_{\text{tok}}$

which depends only on tokens per step, not on model size at all. At 4 M tokens per step that is $1.2\times10^{7}$ FLOP per byte — 600× more than Ethernet requires. DP is not merely tolerable over a slow network, it has enormous headroom, which is why it was the first strategy to work and why it scales to thousands of nodes.

The two clean takeaways:

• TP’s ratio is $H$, a model constant. You improve it only by making the model wider, which is not a knob you have. Hence a hard placement constraint.
• DP’s ratio is $3N_{\text{tok}}$, which you control. Larger global batch, or gradient accumulation, directly buys communication headroom — which is exactly why gradient accumulation is the standard fix when the interconnect limits DP scaling, and why it works when nothing else does.

When memory capacity is the binding constraint and you have bandwidth to spend — which describes most large-model training, and describes it for a reason that is worth spelling out.

What DDP costs in memory. Every device holds a full copy of parameters, gradients, and optimiser state. For Adam in mixed precision that is roughly 16 bytes per parameter: 2 for BF16 weights, 2 for BF16 gradients, and 12 for FP32 master weights plus the two moments. A 70 B model needs about 1.1 TB per device before a single activation exists. There is no GPU with 1.1 TB, so DDP is not a slow option for large models, it is not an option.

What sharding buys and costs. ZeRO’s stages shard progressively — optimiser state, then gradients, then parameters — reducing per-device state by up to the DP degree. FSDP shards all three and reconstructs parameters on demand: all-gather a layer’s parameters before its forward, discard them after, all-gather again for backward (or keep them, per reshard_after_forward), and reduce-scatter the gradients. Total traffic goes from $2P$ per step to roughly $3P$ per step, but spread across every layer instead of one burst.

Why this is a Module 7 decision, exactly. Keeping a parameter shard resident after the forward pass costs capacity and saves a re-fetch; discarding it saves capacity and costs an all-gather. That is keep-resident-versus-refetch, the same trade as a tile in a cache, with the network as the level. reshard_after_forward=True is “evict”; False is “keep”. The right answer depends on whether you are capacity-bound or bandwidth-bound at that moment, and it is legitimately different for different layers — which is why the flag exists per-module rather than globally.

When it is wrong:

• When the model fits comfortably. Plain DDP moves $2P$ once per step and overlaps it with the backward pass almost perfectly. Do not pay 1.5× the traffic and per-layer synchronisation for memory you did not need.
• When the interconnect is slow. FSDP’s all-gathers are on the critical path of every layer’s forward, not a once-per-step background operation. Over a slow network the exposed latency is brutal, whereas DDP’s single all-reduce hides behind a whole backward pass. FSDP wants a fast domain.
• When TP would be better. If you need to shard state anyway, TP shards the compute too, so it reduces per-device FLOPs as well as memory. FSDP replicates the compute and only shards the storage. Inside a node, TP is usually the better use of the bandwidth; FSDP’s advantage is that it composes across nodes where TP cannot go.

The general framing: FSDP converts a capacity problem into a bandwidth problem. That is a good trade exactly when capacity is what you lack, and it is a bad one otherwise — the same statement as every recompute-versus-store decision in the series.

Because it pays in a different currency — idle time and activation memory — and that currency is often more expensive.

The bubble. With $p$ stages and $m$ microbatches, the pipeline takes $p-1$ microbatch-times to fill and the same to drain, giving a bubble fraction of $(p-1)/m$. To get it under 10% you need $m \gtrsim 10p$, which means a large number of microbatches, which means a large global batch — and global batch size is not a free parameter. It is set by optimisation considerations, and beyond some point larger batches stop improving convergence per token. So PP’s efficiency is bounded by a constraint that has nothing to do with systems, which is an unusual and uncomfortable position.

Activation memory. The naive schedule runs all forwards then all backwards, so stage 0 holds activations for all $m$ microbatches simultaneously. This is why 1F1B exists: interleave one forward and one backward so that at steady state each stage holds only $O(p)$ microbatches of activations. Same bubble, dramatically less memory. Interleaved 1F1B goes further by giving each device $v$ non-contiguous chunks of layers, cutting the bubble to $(p-1)/(mv)$ at the cost of $v$× more point-to-point messages.

Load balance. Every stage must take the same time or the slowest sets the rate. Transformer layers are nearly uniform, which helps enormously — but embeddings, the LM head, and the loss are not, so the first and last stages are routinely given fewer layers to compensate. Getting this wrong wastes more than the bubble.

And the real reason it is not the default: PP is the third choice in a composition, not a competitor to the others. The standard 3D arrangement uses all three, each where its cost is cheapest:

Read the table as an assignment problem and it is forced. TP must be innermost because its bandwidth demand can only be met there. DP must be outermost because its headroom is largest and it scales without bound. PP occupies the middle because it is the only strategy whose cost is not bandwidth, which is exactly what you need when crossing the cliff at the chassis boundary. Each strategy is placed where its particular cost is cheapest — which is the same reasoning as binding a tensor to a memory level, one more time.

Staticness — and with it the ability to schedule, overlap and cost the communication at all.

DP, TP, PP and SP all communicate a statically known number of bytes at statically known points. That is what makes them tractable: the collective can be planned at compile time, its buffers pre-allocated, its overlap with compute scheduled precisely, and its cost modelled exactly. Every tool in Parts P and R assumes this.

Expert parallelism does not have it. The all-to-all sends $n_e$ tokens to each expert’s device, and $n_e$ is a random variable set by the router at runtime (Module 9). What breaks:

• Message sizes are unknown until the routing is computed, so buffers must be sized for the worst case or allocated dynamically, and the collective cannot be launched until the gate has run — a serialisation that did not exist before.
• The collective is unbalanced. All-to-all with uneven sizes takes as long as the largest transfer, so a hot expert stalls every device, not just its own. Load imbalance becomes a global synchronisation cost rather than a local inefficiency.
• Overlap is much harder. You cannot start sending before you know what to send, and the computation that would hide the transfer is the computation waiting for it.
• There are two all-to-alls per MoE layer — dispatch and combine — so the cost lands twice, and both are on the critical path.

The mitigations are the same family as Module 9’s, arriving on the network:

• Capacity factor makes the sizes static by padding and dropping. This restores everything — static buffers, planned overlap, exact cost models — and is why essentially every production MoE uses one.
• Load-balancing loss attacks the variance at the source.
• Hierarchical all-to-all: exchange within the node over NVLink first, then across nodes with far less data. The memory-hierarchy principle again, and the largest practical win.
• Overlap the dispatch of one expert group with the compute of another, which is software pipelining across a data-dependent boundary — hard, and where most of the recent engineering effort has gone.

The structural point, and the reason this module ends Part Q: MoE is the first workload where the mapping itself is data-dependent, at the level that is most expensive to get wrong. Sparsity made the indices dynamic; MoE makes the communication pattern dynamic. The whole apparatus — static schedules, compiler-managed memory, analytic cost models, planned collectives — was built on an assumption that MoE denies, and that is why it is simultaneously the most important architecture of the moment and the one that fits the hardware worst. Which is exactly the kind of divergence Module 14 is about.