Three architectures, read as three mappings

The axis is how much uncertainty remains at runtime, and every architectural difference is a consequence of where a design sits on it.

Uncertainty in an ML workload comes from a short list: which addresses will be touched, how long each operation will take, what shapes will arrive, and how much work each unit will get. Someone must absorb each of these — either hardware, at runtime, with mechanisms that cost area and energy on every operation; or the compiler, ahead of time, with analysis that costs nothing at runtime but fails when its assumptions are violated.

Position the four:

• GPU — hardware absorbs nearly all of it. Caches infer residency; warp schedulers hide variable latency by swapping; dynamic allocation and runtime dispatch handle shapes; the work distributor balances load across SMs. Consequence: high control overhead, high latency variance, and a machine that runs almost anything you write.
• TPU — the compiler absorbs residency and most scheduling. VMEM is a scratchpad, so XLA decides every transfer; the systolic array has a fixed dataflow with no runtime choices. Consequence: better energy per MAC, and a recompile whenever shapes change — which is why jit retracing on new shapes is such a visible part of the TPU experience rather than an implementation detail.
• Groq — the compiler absorbs everything, including time. No caches, no dynamic scheduling, every instruction taking a known cycle count, and the compiler emits a cycle-exact timeline for the entire machine including its inter-chip communication. Consequence: latency variance close to zero, which is a genuinely differentiated property; and a compiler that must succeed completely or not at all.
• Cerebras — a different axis, and it is worth being clear about that. Its distinctive move is not static scheduling but removing a memory level: 44 GB of SRAM at ~21 PB/s with no DRAM, so the entire model lives at the top of the hierarchy. It attacks the beachfront limit by declining to have a package boundary in the data path at all.

The three consequences of moving right along the curve, all of which follow rather than being separate design choices:

1. Efficiency improves. Removing tags, schedulers and predictors removes their area and their per-access energy. This is real and it is why the specialised parts have better numbers.
2. The failure mode changes from degradation to a cliff. Dynamic hardware handles the unexpected slowly; static hardware cannot handle it. This is the entire risk of moving right.
3. The compiler’s difficulty grows superlinearly. A GPU compiler that emits a mediocre schedule produces a slow program. A Groq compiler that emits a wrong schedule produces a wrong program. The engineering cost of the right-hand side is mostly software, and it is routinely underestimated — which is Module 14.

It buys the property that the tail equals the mean, which is worth a great deal in a narrow set of situations and nothing at all in the rest.

What determinism means here. No caches, so no cache misses. No dynamic scheduling, so no scheduling variance. No DRAM, so no refresh, no row conflicts, no bank contention. Every instruction takes a known number of cycles, and the compiler emits a static timeline for the whole machine — including chip-to-chip transfers, which are scheduled to arrive on a known cycle rather than synchronised. The result is that a given input takes the same number of cycles every time, and p99 latency equals p50.

What that is worth. ML Hardware Module 5 made the case at length: in a system where you must answer within a deadline, the worst case is the specification, and a mean that is excellent with a p99.9 that is not does not qualify. Eliminating variance rather than reducing it is a categorically different property from being fast on average, and it is why the architecture is interesting for real-time inference and why it produces such striking single-stream token rates — there is no batching required to hide anything, because there is nothing to hide.

What it costs, and the costs are steep:

• No DRAM means capacity is the binding constraint. 230 MB of SRAM per chip means a 70 B model at 8 bits needs several hundred chips just to hold the weights. You are buying capacity in units of entire chips, which is why the economics are dominated by whether the model fits the deployment you can afford. The trade is explicit: SRAM gives you bandwidth and latency at a large multiple of the cost per byte.
• The compiler must succeed completely. A cycle-exact static schedule is not something you can partially produce. Anything the compiler cannot analyse — a data-dependent shape, a dynamic control flow, an unfamiliar operator — is not slow, it is uncompilable. Every item in Part Q is an existential problem rather than a performance problem.
• Utilisation is set at compile time. With no dynamic scheduling, load imbalance cannot be corrected at runtime. If the static schedule leaves a unit idle in some phase, it is idle.
• MoE is exactly the wrong workload. Data-dependent routing, dynamic expert shapes and data-dependent all-to-all sizes are the three things a fully static machine cannot represent. That the field moved toward MoE right as these architectures matured is the single clearest illustration of the timing risk in Module 14.

The fair summary: determinism is a real and rare property, and the architecture that achieves it makes sense for latency-critical inference on models that fit. It is not a general-purpose position on the curve, and its designers do not claim it is. The risk is not that the engineering is wrong — it is that the set of workloads matching its assumptions is chosen by other people.

Because energy per FLOP is a component metric and the decision is made on system and ecosystem metrics, where the advantage shrinks and then reverses.

The advantage is real. No data caches (no tag energy on any access), a systolic array amortising control over 16,384 MACs with PE-to-PE operand movement, no speculation, no dynamic scheduling, and a compiler that plans every transfer. Each of these removes energy that a GPU spends on every operation. On a large, well-shaped, XLA-friendly GEMM the TPU is genuinely more efficient, and Google’s internal economics reflect that.

Why it does not settle the question:

• Achieved utilization, not peak efficiency, is what you pay for. Efficiency at 100% utilization is irrelevant if your workload runs at 30%. The static machine’s utilization is more sensitive to shape mismatch, and the moment the workload has shapes the compiler did not anticipate — a new attention variant, a ragged batch, an MoE layer — the gap narrows or inverts. Module 8’s whole-workload point.
• The remainder is where the time goes. The GEMMs are the efficient part on both machines. Norms, softmax, elementwise chains, data loading, sharding and collectives are the rest, and the advantage there is much smaller because those are memory-bound and both machines are limited by the same HBM physics.
• The compiler is the product. A TPU without XLA is unusable in a way a GPU without a compiler is not, because you can always hand-write CUDA. That means the delivered performance of a TPU is whatever XLA achieves on your model — excellent on the shapes Google runs, and a research project on the ones it does not. There is no escape hatch equivalent to writing a kernel, which is a qualitative difference and not just a maturity gap.
• The ecosystem decides adoption, and it is not close. Almost every model is written against PyTorch and CUDA. Almost every kernel innovation — FlashAttention, paged attention, quantization kernels — lands on CUDA first and is ported later if at all. The specialised part is always some months behind on exactly the techniques that produce the largest wins, and those months are a moving target.

The honest reading is that TPUs are a genuine success by any reasonable standard — they run a very large fraction of one of the world’s largest ML workloads, extremely economically — and that success comes from Google controlling the model, the compiler, the framework and the datacentre, which is to say from internalising the entire codesign loop. That is the real lesson, and it is the strongest argument in this series for why codesign is an organisational question as much as a technical one. A vendor selling a specialised chip to customers who own their own models is trying to do codesign across a boundary where the information does not flow, and Module 14 is about what that costs.

Pick your position on the uncertainty curve deliberately, and then be honest about the two things that actually decide the outcome — neither of which is on your architecture diagram.

On the curve itself:

• Do not sit at the far static end unless you own the workload. The efficiency there is real and so is the cliff, and the cliff is only safe if you control what runs. Google can be static because Google writes the models. A vendor cannot, and pretending otherwise is the most common way these projects fail.
• Be static where the workload has been stable for a decade, dynamic where it has moved recently. Deep reductions over dense tiles have been stable since 2012 — commit to them. Batch composition, sequence length, expert routing and sparsity structure have all moved within one design cycle — leave those to runtime. This is Module 6’s rule with the evidence attached.
• Buy capacity thresholds, not incremental capacity. From Module 7, incremental SRAM buys $\sqrt2$ per doubling; a threshold where the working set fits buys everything. Ask what would have to fit for your target workload, and whether you can get there. If you cannot, spend on bandwidth instead and stop.
• Design for decode, because everyone designs for training and then deploys. Decode is memory-bound with intensity equal to the batch, so bandwidth and capacity matter far more than FLOPs, and a large array actively hurts by raising the batch needed to fill it (Module 12).

And the two things that decide it:

1. Budget the compiler as the primary deliverable. The chip’s delivered value is $\text{peak} \times \text{what the compiler reaches}$, and on a new architecture the second factor starts near 0.2. Every project underestimates this, and it is the single most reliable predictor of whether a good design becomes a good product. If the compiler team is smaller than the RTL team, the plan is already wrong.
2. Ask who else has to change for this to work. If your design requires model authors to adopt a sparsity pattern, a precision format, or a shape constraint, you have a dependency on an organisation with a different objective function. 2:4 sparsity is in every datacentre NVIDIA GPU since Ampere and is little used, for exactly this reason (Module 9). Features that require nobody else to change are worth several times features that do.

The closing judgement, which is uncomfortable and I think correct: the highest-return decision available to most accelerator projects is not architectural. It is to spend more on the compiler, on framework integration, and on making the machine degrade rather than cliff — because those are what convert a peak number into delivered performance, and they are consistently underfunded relative to the silicon they are meant to make usable.