Numerics as a codesign knob

Because the problem is not precision, it is dynamic range, and no redistribution of bits within a 4- or 8-bit element solves it.

An element format spends its bits between exponent (range) and mantissa (precision), and at 4 bits total there is nothing to spend. E2M1 gives you three exponent values and one mantissa bit — six representable magnitudes, positive and negative. A neural network tensor routinely spans several orders of magnitude, with a small number of outlier channels carrying much of the signal. Represent that with a single per-tensor scale and you must choose: scale so the outliers fit and everything else rounds to zero, or scale so the bulk is resolved and the outliers saturate. Both destroy the tensor, and shifting a bit from mantissa to exponent trades one failure for the other.

Block scaling attacks the right variable. Within 32 adjacent elements the spread is far narrower than across the tensor, so a shared exponent per block absorbs the inter-block variation and the element format only has to represent the intra-block variation. The tensor’s global range is carried by the scales; the local detail is carried by the elements. This is why the technique works at 4 bits when nothing else does, and it is also why it generalises — the same reasoning gives you per-channel INT8 quantization, which is block scaling with the block equal to a channel.

The design tension is now explicit and quantifiable:

• Smaller blocks are more accurate — less spread inside a block — and cost more scale storage. 8 bits of scale over 16 elements is 0.5 bits per value, a 12.5% overhead on a 4-bit format. Over 32, it is 0.25 bits, 6%.
• Smaller blocks cost more hardware. The scale must be applied on every block boundary, so smaller blocks mean more scale-application logic per unit of throughput and more scale bandwidth into the MAC array.
• The scale’s own format matters. E8M0 is a pure power of two, so applying it is an exponent addition — nearly free in hardware, and it never introduces rounding error of its own. NVFP4’s E4M3 scale is more expressive (it can scale by non-powers of two, capturing the block’s magnitude more tightly) and requires a real multiply. That is a genuine hardware-versus-accuracy trade, made differently by the OCP consortium and by NVIDIA, and it is the clearest example in the field of two teams solving the same problem with different weights on the same two constraints.

Because the shared scale has to be applied exactly once per block, and the only place it can be applied cheaply is a boundary the datapath already has.

Consider a dot product of length $K$ accumulating in the array. With block scaling, the operands arrive as (element, block-scale) pairs, and the true product of two elements is $(s_a \cdot a) \times (s_b \cdot b) = (s_a s_b)(ab)$. The efficient implementation accumulates $ab$ in the narrow format across a whole block, then applies $s_a s_b$ once and adds into the wide accumulator. That requires the block boundary to coincide with a point where the array naturally breaks its reduction — which is the K-tile depth.

If the block size divides the K-tile, this costs one scale multiply-add per block per output, which is negligible. If it does not:

• You need a divider or a variable-length counter in the reduction control, on every lane. This is not a small amount of logic and it sits in the critical path.
• Partial blocks straddle tile boundaries, so a block’s scale must be carried across a tile boundary, which means extra state and a synchronisation the array was not built for.
• Or you apply the scale per element, which defeats the entire point — the reason block scaling is cheap is that the scale is amortised over the block.

Which is why every shipping format has a power-of-two block size, and why 32 is the consensus: it divides the 128 and 256 depths of systolic arrays, divides the K = 16 tensor core tile in pairs, is a natural fit for a 32-lane warp, and lands in an acceptable place on the accuracy-versus-overhead curve. NVFP4’s choice of 16 buys accuracy by paying more overhead, and it is viable precisely because NVIDIA controls both the format and the array and could co-design the two.

The general principle, which is the reason this module is in the series: a numerical format is a hardware interface. It is negotiated against the array’s reduction depth, the memory system’s access granularity, and the register file’s width — and a format designed without those constraints will either not be implemented or will be implemented badly. The MX specification coming out of a hardware consortium (AMD, Arm, Intel, Meta, Microsoft, NVIDIA, Qualcomm) rather than from a numerics group is not an accident of politics; it is what the problem requires.

Because the error in a dot product accumulates over $K$ terms while the error in a single element does not, and $K$ is thousands.

The mechanism is straightforward. Each addition into the accumulator rounds. With $K$ sequential additions in a format with unit roundoff $u$, the worst-case relative error grows like $Ku$ and the random-walk expectation like $\sqrt K u$. For $K = 4096$: in FP16 ($u \approx 5 \times 10^{-4}$) the expected relative error is around 3%, which is not usable. In FP32 ($u \approx 6 \times 10^{-8}$) it is around $4 \times 10^{-6}$, which is fine. There is also a second, sharper failure mode — swamping — where the running sum grows large enough that small individual terms round to nothing when added, so they contribute exactly zero. In a narrow accumulator with a long reduction, a substantial fraction of the terms can simply be discarded, and the result is not noisy, it is systematically wrong.

The consequences, which are exactly the ones people forget when they estimate quantization speedups:

• The accumulator is not quantized, so its cost does not fall. Register pressure, accumulator bandwidth and the psum traffic of Module 6 are unchanged when you go from BF16 to FP8. You halve the operands and nothing else.
• The MAC’s real shape is narrow × narrow → wide. Which is why FP8 gives 2× rather than 4× on many designs: the multiplier shrank quadratically, the adder tree and accumulator did not.
• Split-K makes it worse, because each partial reduction is combined again afterward, adding another rounding level — and combining in a narrow format is where multi-pass reductions actually lose accuracy.

The mitigations worth knowing:

• FP32 accumulate is the default and should stay so. The cost is real and the alternative is unreliable. Tensor cores accumulate in FP32 even for FP8 inputs, deliberately.
• Hierarchical accumulation. Accumulate in a narrow format within a block, promote to wide at block boundaries. This bounds the error to $\sqrt{\text{block}}$ rather than $\sqrt K$, and it is another reason blocks are useful beyond scaling.
• Stochastic rounding, primarily in training, which makes the accumulation error zero-mean rather than biased and lets narrower accumulators survive longer. Cheap in hardware (an RNG per lane) and increasingly common.
• Kahan or compensated summation where accuracy matters more than throughput — rare in ML, standard in HPC, and worth knowing exists.

The framing to keep: precision is not one decision, it is one decision per tensor per role. Operands can be 4 bits because they are read once each; accumulators cannot, because they are touched $K$ times. Any statement about “the precision of the model” that does not distinguish the two is not saying anything.

Spend it wherever the binding constraint is, and recognise that the answer differs per phase of the same workload — which is why a single global precision setting is always leaving something on the table.

Work the cases:

Decode-phase LLM inference. Bound by memory bandwidth: every token reads the entire weight set, and the arithmetic intensity is roughly 2 FLOP per weight byte at batch 1 (Module 12). So spend the budget on weight bit width, and spend it aggressively — 4-bit weights give close to a 4× token rate against BF16, and it does not matter whether the hardware has a native FP4 MAC, because dequantizing to BF16 before a BF16 tensor core still delivers the full memory win. This is the case where “the format is emulated” is not an objection. The KV cache is the second target for exactly the same reason, and it is nearly free in accuracy terms. Activations barely matter: there is one token’s worth of them.

Prefill and training. Compute-bound, so spend the budget on the operand format the MAC natively supports. FP8 for the GEMM operands is the live choice; going narrower buys nothing unless the array supports it natively, since an emulated format costs unpack work on a compute-bound kernel. Weight footprint matters much less here because weights are reused across the whole batch.

Distributed training. Often bound by the interconnect on the gradient all-reduce or the tensor parallel activations. Spend on the wire format — BF16 or FP8 gradients — which costs nothing in compute or memory and directly halves the bytes on the scarce resource. Note this is a place where the accuracy risk is unusually well understood, since gradient noise is already large relative to quantization noise.

Never spend it on: the accumulator, the master weights during training, or the softmax and normalisation statistics. These are cheap in bytes and catastrophic in error, and they are where narrow-precision efforts most often fail for reasons the team then misattributes to the operand format.

The decision procedure, compactly:

1. Identify the binding resource for the phase in question.
2. Choose the tensor whose bytes flow through that resource most.
3. Reduce its precision, and only its precision.
4. Verify against accuracy, then re-identify the binding resource — because you have probably moved it, and the next reduction should target somewhere else entirely.

Step 4 is the one that gets skipped, and it is why quantization efforts frequently show a big first win and nothing afterwards: the second change was aimed at the resource that stopped binding after the first one.