3. What graph mode actually buys — ML Software & Compilers

The core mental model

Graph mode attacks exactly three overheads, and it is worth separating them because they have different sizes, different mechanisms, and different conditions under which they bind.

The three overheads, and the fact that they bind in different regimes. Fusion pays where the work is elementwise and memory-bound; launch elimination pays where the ops are small and numerous; memory planning pays where allocation churn is the problem. A model already dominated by a few large cuBLAS matmuls has none of the three, which is why its speedup is about 1.0×.

Launch overhead is the per-op CPU cost from Module 2, around 5–10 µs. Capturing a graph lets you issue the same sequence without re-running Python, the dispatcher and autograd every iteration; CUDA Graphs (Module 9) go further and collapse thousands of launches into one replay. This dominates when kernels are small — batch 1, small models, decode.

Memory traffic is the larger prize on most real workloads. Eager mode materialises every intermediate to HBM and reads it back for the next op, so a chain like x.mul(a).add(b).relu() moves the tensor across the memory bus six times for three trivial arithmetic operations. Fusing them into one kernel moves it twice. In the hardware series’ terms this is a pure arithmetic intensity win: identical FLOPs, a third of the bytes. This dominates whenever you have long chains of elementwise or reduction work, which is to say most models outside the matmuls.

Allocation is the smallest of the three but not zero: one allocator request per intermediate, and with a graph you know every buffer’s lifetime in advance and can plan reuse (Module 8).

The practical consequence is a diagnostic rather than a promise. torch.compile is not a uniform speed-up; it is three mechanisms whose value depends entirely on which ceiling you are against. A model already dominated by a few large cublas matmuls has no launch overhead worth removing and nothing to fuse, so compiling it yields nearly nothing — and that is not a failure, it is the correct answer. Measure which overhead binds before compiling, and you can predict the result instead of discovering it.

How it actually works

The three overheads, with the condition under which each one actually binds:

Overhead	Size	Mechanism	Binds when
Launch	5–10 µs/op CPU	graph capture, CUDA Graphs	small kernels, batch 1, decode
Memory traffic	up to $N{\times}$ on a chain of $N$	fusion	elementwise/reduction chains
Allocation	0.3–1 µs/tensor	compile-time planning	many small intermediates

The fusion arithmetic, for a chain of $N$ elementwise ops on a tensor of $B$ bytes:

	Bytes moved	For $N=3$
Eager, unfused	$2NB$ (read + write each)	$6B$
Fused	$2B$ (read once, write once)	$2B$
Ceiling	$N\times$	3×

Typical measured torch.compile speed-ups, and why they differ so much:

Workload	Speed-up	Which overhead
Small model, batch 1 inference	1.5–3×	launch
Transformer training, large batch	1.1–1.4×	fusion in the non-matmul work
Elementwise-heavy (norms, activations)	1.5–2×	fusion
Already-fused big GEMMs (`cublas`)	~1.0×	none — nothing left
Diffusion U-net inference	1.3–1.8×	fusion + launch

Cost of compiling	Value
First-call compile, medium model	30 s – 5 min
`mode="max-autotune"`	several × longer
Warm start (Inductor cache)	seconds
Recompile on shape change	full cost again (Module 10)

Critical thinking

You compile a model and get no speed-up. Work out why, without guessing.

Establish which ceiling binds — the same partition as Module 2 plus one more term.

Was it launch bound? Sum self CPU against self CUDA in the profiler. If CUDA already dominated, there was no launch overhead to remove and one of the three mechanisms was already irrelevant.
Was there anything to fuse? Count kernels before and after. If the op mix is a handful of large cublas GEMMs, they were already single fused kernels written by people who do nothing else, and Inductor will correctly not touch them. No fusion opportunity means no fusion win.
Did it actually compile? TORCH_LOGS=graph_breaks — a model shattered into many small graphs keeps most of its overhead, and this is the most common real cause (Module 4).
Is it recompiling every step? TORCH_LOGS=recompiles. Varying shapes can mean you pay compilation repeatedly and never reach steady state (Module 10).
Are you measuring steady state? Warm-up, and torch.cuda.synchronize() before timing, or you are measuring queue depth rather than work.

The outcome worth accepting: for a large-batch GEMM-dominated model, ~1.0× is the right answer. The mechanisms attack overhead, the workload had none, and no amount of tuning changes that. Knowing this in advance is more valuable than the speed-up would have been.

Why does fusion help even when it does not reduce FLOPs at all?

Because on a memory-bound op, time is set by bytes moved, and FLOPs are not the currency.

Take y = relu(x * a + b) on a tensor of $B$ bytes. Eager runs three kernels: multiply reads $B$ and writes $B$ ; add reads $B$ and writes $B$ ; relu reads $B$ and writes $B$ . Six $B$ of HBM traffic for three arithmetic operations per element. Fused: read $x$ once, do all three operations in registers, write once. Two $B$ . Three times less traffic, identical arithmetic.

Two refinements that matter in practice. The intermediates may be L2-resident — on an H100 with 50 MB of L2, a tensor under ~25 MB never reached HBM in the first place, so the fusion win at the HBM boundary is smaller than the naive count suggests, though you still save the launches and the L2 traffic. And fusion raises register pressure, so a very long fused chain can spill or drop occupancy, at which point the fused kernel achieves lower effective bandwidth. Inductor’s scheduler is making exactly this trade when it decides where to cut a fusion group (Module 6).

The general statement: fusion’s ceiling is bytes-before over bytes-after at whichever memory boundary actually binds. That is why it is worth so much on normalisation and activation chains — which are pure memory traffic — and worth nothing on a big GEMM, which was already compute bound.

Given a model you have not profiled, predict whether compiling will help.

Three questions, answerable from the architecture alone.

What is the batch size and the kernel size? If tensors are small enough that per-op GPU time is in the low microseconds, launch overhead dominates and compilation will help substantially. Batch-1 inference of anything under a few billion parameters is in this regime.

What fraction of time is in matmuls versus everything else? For a transformer, the matmuls are maybe 60–80% of FLOPs but often only 40–60% of time, because normalisation, activation, residual adds, dropout, masking and the softmax are memory bound and unfused. That non-matmul remainder is what fusion attacks, and it caps your win: if 40% of time is fusible and fusion makes it 3× faster, Amdahl gives $1/(0.6 + 0.4/3) \approx 1.36\times$ . That number matches the measured 1.1–1.4× for transformer training closely enough to be a genuinely useful prediction.

How dynamic is it? Varying sequence lengths, MoE routing, data-dependent control flow — each pushes toward recompilation or graph breaks, and the expected value falls (Parts M).

The point of doing this before compiling is that it converts a 30-minute experiment into a two-minute estimate, and more importantly it tells you which knob to reach for when the answer is disappointing.

When is compiling a net loss?

Four situations, and they are common enough to be worth naming.

Compile time exceeds the run. A 2-minute compilation to save 10% on a 30-second job is arithmetic that does not work. This bites hardest in development, where you run something once.
Perpetual recompilation. Shapes that vary continuously with a cache-size limit reached mean you pay compilation forever and may end up worse than eager. Module 10.
Debuggability. A stack trace through generated Triton is worse than one through eager. During development the right call is often to leave it off and enable it when shipping.
Numerics drift. Fusion changes the order of floating-point operations, so results differ in the last bits — and reductions reassociated during fusion can differ more than that. Usually harmless; occasionally the cause of a failing regression test, and always the cause of confusion when nobody expected it.

The framing carried from the hardware series: compilation is an optimisation with a cost, not a free improvement. Apply it where the overhead it removes actually exists, and measure both the steady-state gain and the compile-time price.

Self-check

Name the three overheads graph mode attacks, their sizes, and when each binds.

Launch overhead, 5–10 µs of CPU per op, removed by capture and graph replay — binds when kernels are small, so batch 1 and decode. Memory traffic, up to $N\times$ on a chain of $N$ elementwise ops, removed by fusion — binds on normalisation/activation chains. Allocation, 0.3–1 µs per tensor, reduced by compile-time planning — binds when there are many small intermediates.

Compute the fusion ceiling for a 4-op elementwise chain, and give two reasons you will not reach it.

Eager moves $2NB = 8B$ ; fused moves $2B$ ; ceiling 4×. You will not reach it if the intermediates were L2-resident (under ~25 MB on an H100 they never touched HBM), or if the long fused chain raises register pressure enough to spill or cut occupancy, lowering achieved bandwidth. Inductor’s scheduler is trading exactly these when it decides where to break a fusion group.

Predict the speed-up for transformer training where 40% of time is fusible non-matmul work.

If fusion makes that 40% about 3× faster, Amdahl gives $1/(0.6 + 0.4/3) \approx 1.36\times$ — which matches the measured 1.1–1.4× for transformer training. The matmuls are already single fused cublas kernels, so there is nothing there to win; the entire gain comes from the memory-bound remainder.

You compiled and got 1.0×. Give the five checks, in order.

Was it launch bound (sum self CPU vs self CUDA)? Was there anything to fuse (kernel count before and after; large cublas GEMMs offer nothing)? Did it actually compile (TORCH_LOGS=graph_breaks)? Is it recompiling every step (TORCH_LOGS=recompiles)? Are you measuring steady state (warm-up plus cuda.synchronize())? For a large-batch GEMM-dominated model, 1.0× is the correct answer, not a failure.

Give four situations where compiling is a net loss.

Compile time exceeding the run itself, which bites in development; perpetual recompilation from continuously varying shapes; degraded debuggability, since stack traces run through generated Triton; and numerics drift, because fusion reorders floating-point operations and reassociates reductions, which is usually harmless but reliably surprising when a regression test fails.