Compute: the shape of the multiply

They are two answers to “how do I amortise the cost of controlling a multiplier”, and they amortise different things.

The systolic array amortises control and operand movement spatially. Weights are loaded into the PE grid and stay there; activations flow in from one edge and partial sums flow out the other, with each PE passing values to its neighbour. One control decision covers 16,384 multipliers, and — the part people miss — an operand fetched once from SRAM is used by an entire row or column of PEs through nearest-neighbour wires that are a few microns long. That is why the TPU’s energy per MAC is excellent: it has almost no instruction overhead and almost no long-wire operand traffic. The cost is rigidity. The array has one dataflow, cast in RTL, and the shape it wants is the shape you must provide.

The tensor core amortises less, and keeps flexibility. It is a matrix unit sitting inside a SIMT core, fed from the register file, issued as an instruction by a warp scheduler. The operand shape is much smaller (m16n8k16 versus 128×128), so the problem sizes that fill it are far smaller, and the surrounding machinery — predication, dynamic scheduling, address generation, a general memory hierarchy — means anything you can express in CUDA still runs. You pay for that in register file bandwidth, which becomes the real constraint (a tensor core operation reads its operands from the RF every time, rather than from a neighbour), and in instruction issue, which is why Hopper added warp-group-level operations and asynchronous copies to cut the per-operation overhead.

Concretely, the trade is:

The interesting convergence is that both sides have been moving toward the middle. NVIDIA keeps enlarging the effective operation (warp-group MMA, tensor memory accelerator, distributed shared memory across an SM cluster) because the systolic-style amortisation is genuinely more efficient, and TPUs keep adding vector units and scalar cores beside the MXU because pure systolic cannot express everything a real model needs. Nobody thinks the endpoints are right.

Compute achieved arithmetic intensity and compare it against the machine’s ridge point. That single comparison separates the cases, and it must be achieved intensity — actual bytes moved from HBM, from a profiler — not the algorithmic ideal.

Starved shows up as achieved intensity below the ridge, with measured DRAM bandwidth near peak. The machine is doing exactly what it should; you asked for too many bytes per FLOP. The fixes are all mapping fixes: tile larger so operands are reused more before eviction, fuse adjacent operations so intermediates never reach HBM, change layout so accesses coalesce into full bursts, or reduce bytes with a smaller dtype. Buying a chip with more FLOPs does nothing whatsoever.

Shape-mismatched shows up as intensity comfortably above the ridge with memory bandwidth mostly idle — the operands are there, and the arithmetic units are still not full. Now look at the tile dimensions against the native operation shape. The usual culprits are a small dimension (a GEMM with N = 3 on a unit that wants multiples of 8 or 128), a K dimension shorter than the array’s reduction depth, or a batch too small to fill the spatial assignment. The fixes are structural: batch more requests, pad and accept the waste, choose a different loop dimension to map across the array, or fuse several small operations into one large one (which is exactly what grouped and batched GEMM APIs exist to do).

Two refinements worth having, because the clean dichotomy fails in real profiles:

• Latency-bound is a third state and it looks like both. Neither compute nor bandwidth is saturated, because there is not enough work in flight to cover memory latency. The fix is more parallelism or deeper prefetching — more warps, deeper software pipelining, asynchronous copies — not tiling and not batching.
• They interact. Raising intensity means bigger tiles; bigger tiles consume registers and shared memory; that can reduce the number of concurrent thread blocks below what is needed to hide latency, converting a starved kernel into a latency-bound one. This is why tile-size tuning has a sweet spot rather than a monotone direction, and it is the practical reason Module 8’s search exists at all.

Because precision is the one knob that moves all three resources simultaneously, and because the overheads it introduces are not on the same axis as the gains.

Why more than 2×. Halving the bit width halves memory footprint and halves the bytes moved at every level of the hierarchy — HBM traffic, cache pressure, interconnect payload — at the same time as it doubles arithmetic throughput. For a memory-bound kernel the FLOP/s doubling is irrelevant and the traffic halving gives you the full 2× on its own. For a compute-bound kernel you get the arithmetic doubling and an improved ridge-point position. And there are second-order wins that are often the largest ones in practice: a model that now fits in HBM instead of spilling, a KV cache that fits so batch size doubles, or weights that fit in SRAM entirely, which is a change of memory level rather than a change of degree.

Why sometimes far less. The gains are only realised if the format is native to the compute unit and the whole pipeline stays in it. The failure modes:

• Emulated formats. INT4 weights dequantized to BF16 before a BF16 tensor core sees them buys you memory traffic and nothing on compute — often the right trade for decode, but not a 2× on FLOPs.
• Conversion overhead. Pack, unpack and scale application are real vector work. On a kernel that was already memory-bound this is free (it hides), and on a compute-bound one it eats the gain.
• Accumulation is not quantized. You multiply in FP8 and accumulate in FP32, so the accumulator traffic and the register pressure are unchanged. Speedups on the multiply do not extend to the reduction.
• Everything else in the model. Amdahl again: quantize the GEMMs and the softmax, layer norms, activations and elementwise ops are untouched. In a transformer these are a small share of FLOPs and a large share of memory traffic, so they cheerfully become the new bottleneck.

The general form, which is worth carrying into Module 10: a precision change is not “a speedup”, it is a simultaneous move on three axes with a fourth axis — accuracy — as the cost. Whether it pays depends entirely on which resource you were bound by before you made it, and the same change is a 4× win in decode and a rounding error in prefill.

Because it is the only number that is comparable across vendors without disclosing anything, and because the alternatives are all workload-dependent in ways that make them arguable.

There is no honest way to compare an H100 and a TPU v5p on “effective throughput” without agreeing on a model, a batch size, a parallelism strategy, a software stack and a measurement methodology — and every one of those is a place where the benchmark can be shaped. Peak FLOP/s is arithmetic on a datasheet: multipliers × clock × operations per multiplier. It cannot be gamed except by counting things that should not be counted, which is exactly what happens.

The standard distortions, worth being able to spot instantly:

• Sparsity multipliers. Quoting the 2:4 structured-sparse rate as if it were the dense rate doubles the number for a feature most workloads do not use (Module 9). The H100’s “1979 TFLOP/s” is this — the dense BF16 figure is 989.
• Lowest-precision quoting. Advertising the FP4 rate against a competitor’s BF16 rate is a 4× free lunch on a slide.
• Boost clocks that no sustained workload holds under power and thermal limits.

The metric that actually replaced it inside serious teams is MFU — model FLOP/s utilization — achieved useful FLOP/s over peak, on your real model. It is honest because it prices in shape mismatch, starvation, communication stalls and pipeline bubbles all at once, and the numbers are sobering: large-scale LLM training typically lands in the 35–50% range, and decode-heavy inference frequently sits in the single digits, because decode is structurally memory-bound and no amount of compute helps.

Which is the argument for the whole series in one number. If a well-engineered system reaches 40% of peak, then the mapping is worth more than the remaining 60% of a hardware generation — and it is also why a vendor’s peak and your throughput can move in opposite directions across a generation, as larger arrays raise the peak while making the filling condition harder to meet.