The codesign loop, and where it fails

By separating the parts of the workload that have been stable for a decade from the parts that have moved within one design cycle, and committing only to the first.

What has been stable since 2012, and is therefore safe to build into silicon:

• Deep reductions over dense rectangular tiles. Every dominant architecture — conv, transformer, MoE expert — is a GEMM at its core.
• The memory hierarchy’s shape, and the fact that data movement dominates energy. This is physics, not fashion.
• Reduction depths in the thousands, which is what makes output-stationary accumulation and FP32 accumulators the right answers.
• The need for higher arithmetic intensity every generation, since the beachfront limit is geometric.

What has moved within a single design cycle, and must therefore stay soft:

• Batch composition and sequence length. Long context changed KV assumptions in about two years.
• Sparsity structure. A decade of unstructured-pruning hardware proposals, and the sparsity that matters turned out to be routing.
• Routing and conditional computation. MoE went from research curiosity to dominant in roughly three years.
• Numerical format. FP16 to FP8 to block-scaled FP4 inside four years — which is exactly why a datapath with a parameterised scale application survived and a fixed-format one did not.
• The exact operator set. Every new attention variant is an operator nobody had when the chip was designed.

Concrete tactics that follow:

• Parameterise anything cheap to parameterise. A configurable block-scale size costs almost nothing and covers three format generations. A fixed one covers your guess.
• Prefer capacity over specialised structure. SRAM is useful for whatever arrives; a sparse-operand-matching unit is useful for one thing that may not.
• Ask what happens at 5× off-target. A design that degrades to 60% on an unanticipated shape is a good bet; one that degrades to 5% because the array cannot fill is a bad one at the same expected value, because the downside is unrecoverable and the loss distribution is what matters, not its mean.
• Build the escape hatch. A machine that permits hand-written kernels can be rescued by a motivated user when the compiler falls short. A machine that can only run what its compiler emits cannot. This is worth more than it looks, because the rescues that matter are the ones nobody planned.

And the meta-tactic: shorten the loop. Chiplets let you re-spin one die rather than a whole design. FPGA and emulation prototypes let the compiler team start eighteen months earlier. A programmable rather than fixed-function unit converts a hardware bet into a software one. Every month removed from freeze-to-production is a month less drift to predict, and that is a more reliable improvement than being better at predicting.

Because its contribution is invisible in every artefact the organisation uses to make decisions, and because it is structurally positioned as a cost rather than as the product.

The mechanisms, and they are consistent enough across projects to be predictive:

• The datasheet has no compiler column. Peak FLOP/s, memory bandwidth and capacity go on the slide. “Achieves 45% MFU on your model” does not, because it depends on the model. So the thing that determines delivered value is invisible in the artefact used to sell, compare and fund the project.
• Hardware has an unmissable deadline and software does not. Tape-out is a date with an enormous cost attached. Compiler maturity is a curve with no cliff, so when schedules compress, the resource that moves is the one without a hard date. Every project does this and every project regrets it in bring-up.
• The counterfactual is unobservable. If the compiler reaches 40% of peak, nobody sees the 65% a better one would have reached. There is no experiment that reveals the gap, so it is never priced. Meanwhile a hardware shortfall is measurable against the datasheet and gets attention immediately.
• The specialisation trap. The more specialised the hardware, the more the compiler must do — and the more novel it must be, so less can be reused from LLVM or MLIR. Effort scales with specialisation in exactly the direction that makes the estimate worst. Teams that budget the compiler by analogy to a GPU backend, on hardware far more static than a GPU, are wrong by a large factor.
• The hiring pool is small and the work is invisible. Compiler engineers who understand accelerators are scarce, and the achievement — “we reached 55% instead of 35%” — is much harder to narrate than a chip.

The numbers make the argument. From the table above, three of the five loss terms between peak and delivered MFU are compiler-owned, together worth roughly 1.3–2.5×. A hardware generation buys 1.5–2×. So the compiler gap is the size of a hardware generation, on silicon you already own, and it is available without a tape-out, a new process node, or a two-year wait.

The organisational fix is not subtle, which makes the failure more striking: put the compiler under the same leadership as the architecture, staff it comparably, start it before RTL freeze, and — most importantly — make the mapping study and the compiler the same team’s work. The people who chose the capacities from a model should be the people who then have to hit those numbers on silicon. When those are different teams, the model’s optimism is never corrected, because nobody who suffers from it had the chance to object.

Aged badly: convolution-specific dataflows. Row-stationary (Module 6) and the wave of accelerators built around convolution’s sliding-window reuse were correct engineering on the workload of 2016. Their entire value came from a reuse structure — overlapping windows — that transformers do not have. When the workload moved, the specialisation did not become less valuable, it became worthless, because the structure it exploited was absent rather than diminished. A comparable fate met the unstructured-sparsity accelerators, which bet on a model property that model teams never had sufficient reason to deliver.

Aged well: the systolic array and matrix units generally. The TPU’s MXU bet on deep reductions over dense rectangular tiles, which was true of convolution in 2015, true of transformers in 2020, and true of MoE experts today. NVIDIA’s tensor cores made the same bet at a smaller granularity. Nine years later the bet is still paying, and the reason is that it was made on the loop nest rather than on the model.

The distinguishing question is exactly that: is the specialisation aimed at the shape of the computation, or at a property of the current model family?

Two refinements worth having, because the clean dichotomy is too comfortable:

• Some model-family bets pay off enormously anyway, if the family lasts long enough to amortise the chip. Betting on transformers in 2020 was a model-family bet and it was correct for five years, which is longer than a chip’s commercial life. The bet is not wrong because it is a bet — it is wrong when its horizon is shorter than your design cycle. That is a checkable condition and most teams never state it.
• The failure mode matters more than the probability. A loop-nest bet that is wrong leaves you with a slower general machine. A model-family bet that is wrong leaves you with silicon that cannot run the workload at all. Given a 2.5–4 year freeze-to-production window and a 3-year historical drift interval, you should be pricing a substantial probability of being wrong and choosing bets whose downside is survivable.

The one-line test: if the model architecture you designed for stopped existing tomorrow, what fraction of your chip would still be useful? For a matrix unit with a large SRAM, nearly all. For a convolution dataflow with a flexible NoC, almost none. Both cost the same to design.

You cannot optimise a mapping onto hardware you did not design, and you cannot design hardware without assuming a mapping. They are one problem, and every organisation that treats them as two ships a chip whose peak numbers nobody can reach.

The argument, assembled from the whole series:

The abstraction is three resources — compute, memory, interconnect — each exposing capacity, bandwidth, latency, granularity, and the field that makes it codesign: who decides placement (Module 1). Compute’s real interface is the shape of its native operation and the condition under which it fills, not its FLOP/s (2). Memory is a ladder whose rungs differ in granularity and in whether the hardware or the compiler owns residency (3). The interconnect is the same ladder continued past the package, with topology in place of capacity and a hard cliff at a boundary set by geometry rather than engineering (4).

The act that binds them is mapping: tiling, permutation, spatial assignment, binding (5). From which the central consequence follows — arithmetic intensity is a property of the mapping, not the algorithm — and therefore neither side of the interface can be optimised alone. Dataflow is that choice cast in metal, and casting it fixes one dimension of the mapspace for the life of the chip (6). Traffic falls only as the square root of capacity, so incremental memory is weak and thresholds are everything (7). And because a mapping study runs on hardware that does not exist, the cost model’s accuracy is the chip’s accuracy (8).

Then the three things that break it: irregularity, where hardware exploits sparsity only if the pattern was shaped to the array (9); precision, the one knob that moves all three resources, whose block size must divide the array’s K-tile (10); and scale, where parallelism turns out to be the same four decisions with the network as the memory level (11). Attention demonstrates all of it at once — FlashAttention as a re-binding, prefill and decode as opposite points with identical mathematics, and GQA as the model bending to a memory constraint (12). And the architectures differ mainly in who absorbs uncertainty at runtime, with the GPU winning not on efficiency but on never falling off a cliff (13).

Which leaves the practical conclusions, in the order I would act on them:

1. Find the binding resource before optimising anything, and expect the answer to be unglamorous — an allocator, a shape, a collective.
2. Fix the numbers, keep the mapping soft. Commit resources; refuse to commit how work is assigned to them.
3. Budget the compiler as the primary deliverable. Three of the five loss terms between peak and delivered performance belong to it, worth a hardware generation, available without a tape-out.
4. Prefer designs that degrade over designs that cliff, because the workload will move and the loss distribution matters more than its mean.
5. Close the loop organisationally, or accept that you are doing half of codesign. The information has to flow from the workload back to the architecture, and it does not cross a company boundary.

And the caveat I would keep attached to all of it: this framework is a good description of dense tensor computation on the hardware of the last decade, which is precisely the regime where the assumptions hold. The workloads currently straining it — data-dependent routing, dynamic shapes, irregular sparsity, agentic and heterogeneous inference — are straining the framework, not just the hardware, because they break the static analysability everything here rests on. If that continues, what changes is not the answers in this series but the questions, and the next version of this argument will be about how to reason when the mapping itself is a runtime value.