The codesign contract

Because codesign requires two conditions simultaneously, and most domains have at most one.

The interface has to leak. For codesign to be worth anything, performance must depend on decisions that cross the hardware/software boundary. In ML it does, absolutely: the cost of a convolution varies by an order of magnitude depending on tiling and layout, and neither side can compute it alone. The hardware designer cannot say how much SRAM is enough without assuming a tiling; the compiler cannot choose a tiling without knowing the SRAM. That is a genuine circular dependency, and circular dependencies between teams are exactly what codesign exists to resolve.

The workload has to be narrow and stable. Specialisation is a bet that the thing you specialised for still exists when the silicon arrives, two to three years later. Deep learning is an extraordinarily good bet by this standard: GEMM, convolution and attention are all the same shape — a perfectly nested affine loop over dense rectangular tensors with a reduction — and that has been stable since roughly 2012 even as architectures churned from AlexNet to ResNets to transformers to mixtures of experts. The models changed constantly; the loop nest barely moved.

Now apply the test elsewhere. A web server’s interface leaks too, but the workload is a sprawl of string handling, syscalls, TLS and branchy business logic with no dominant kernel to specialise for — narrowness fails. Databases are closer than people think, which is exactly why you do see specialisation there, in the narrow parts: hardware compression, encryption, checksum offload, and SmartNICs. The pattern holds. Specialisation follows narrowness wherever it appears.

The instructive negative case is graphics before shaders. Fixed-function pipelines were the right answer while the workload was narrow, and became the wrong answer the moment programmability mattered more than efficiency. That is the risk you take every time, and Module 14 is about what happens when the bet loses.

For a single workload at a single operating point, yes — and that is why “balanced design” is a harder idea than it sounds.

Take one kernel. It has one binding constraint. If a decode-phase GEMV is memory-bandwidth-bound, then compute beyond what saturates the bandwidth produces exactly zero additional throughput. Adding FLOPs is not slightly wasteful, it is entirely wasteful. This is just Amdahl, and it is why “we doubled the tensor cores” so often shows up as a single-digit end-to-end gain.

The subtlety is that a chip does not run one kernel. It runs a mix, and different members of the mix bind on different resources — prefill is compute-bound, decode is memory-bound, all-reduce is interconnect-bound, and they can be the same model in the same request. So balance is a property of a design against a workload distribution, not against a workload. The design question is not “which resource is the bottleneck” but “what fraction of my expected work, weighted by how much I care about it, binds on each resource, and does the marginal transistor buy more in one place than another”.

Three consequences that follow, and are easy to miss:

• The mix is not fixed, because the mapping is a free variable. If you are memory-bound, a better mapping may make you compute-bound, at which point the compute you “wasted” money on starts paying. Provisioning is a bet on the mappings your compiler will eventually find, not on the ones it finds today.
• Underprovisioning is asymmetric with overprovisioning. Too much compute is idle silicon: you paid area and leakage. Too little memory bandwidth is a hard ceiling that no software can lift for the chip’s entire life. Given uncertainty, err toward the resource software cannot work around.
• Idle resources still cost the thing you actually care about. Area, power budget and yield are shared. Unused tensor cores are not free — they lower clocks through power limits and take die space that could have been SRAM, which is the resource that would have helped.

The sharp version: a balanced chip is one where no resource is the bottleneck for a workload you care about that you cannot re-map away from. Both qualifiers are doing work.

Because it is not an engineering oversight, it is what the physics of wires does under scaling, and it has been getting worse rather than better for twenty years.

Transistors got cheaper, faster and lower-energy with each process node. Wires did not. The energy to move a bit is roughly proportional to the capacitance you must charge, which is proportional to the distance travelled, and shrinking a process does not shorten the distance from the edge of a die to a DRAM stack. So every node makes arithmetic cheaper while leaving off-chip movement roughly where it was. The ratio in the table above is not a snapshot; it is a trend line, and it has been widening the entire time.

What has actually been done about it, all of which is reducing distance rather than making distance cheap:

• Bring memory closer. HBM is DRAM stacked next to the die on an interposer rather than on a DIMM across a board — the same bits, a much shorter wire, which is why HBM’s energy per byte is several times better than DDR at far higher bandwidth. This is the single largest win available and it has been almost entirely spent.
• Move data less, by keeping it still. Every dataflow and tiling technique in Part P. If a value crosses the expensive boundary once and is used a thousand times, you have amortised the 640 pJ. This is the lever software controls, and it is the reason mapping is worth this much attention.
• Move fewer bytes. Quantization, which is Module 10 — the only knob that reduces traffic on all three resources at once.
• Do not move it at all: processing-in-memory. Genuinely attacks the root cause, and has been ten years away for about twenty years, because DRAM processes are terrible for logic and because the programming model is a research problem rather than a product.

The honest summary is that the ratio is a boundary condition rather than a bug. It is why the three-resource abstraction is the right one — if arithmetic were the cost, you would only need to count FLOPs, roofline would be a straight line, and none of this would be interesting.

It relocates a decision between two parties with completely different information, and that is the central trade of the series.

Hardware decides (a cache). The hardware observes the actual access stream and reacts to it. It has perfect information about what did happen and none about what will. It costs area and energy — tags, comparators, replacement state, coherence — and its behaviour is only statistically predictable, which is precisely the property that makes it unusable in a hard-deadline system. But it never falls off a cliff, because an unanticipated access pattern still works, just slower.

Software decides (a scratchpad). The compiler knows the whole loop nest ahead of time, so for an affine, statically-shaped computation it can compute the optimal residency schedule. No tags, no tag energy, no misses, deterministic timing, and every byte of the array is usable. The catch is total: whatever the compiler failed to anticipate does not degrade gracefully, it fails. Dynamic shapes, data-dependent control flow and irregular access are not slow on a scratchpad, they are unrepresentable, and the compiler must fall back to something conservative.

ML is unusually favourable to the software side, which is why explicit management keeps winning ground: the loop bounds are known, the access patterns are affine, and the reuse is enormous and statically analysable. That is why TPUs use compiler-managed VMEM rather than caches, and why the GPU trend runs the same direction — shared memory has always been a scratchpad, and Hopper’s TMA is hardware for making the compiler’s explicit schedule cheap to express.

The general principle, which shows up again in Modules 3, 8 and 13: hardware management is insurance you pay for continuously, and its premium is worth it exactly in proportion to how much uncertainty is left at runtime. ML has less than almost anything else, so it buys less insurance — until the workload turns dynamic, at which point Part Q is about the bill arriving.