14 modules · 5 parts · ~221 min
ML Software & Compilers
Fourteen modules on the stack between model code and the GPU. Why eager execution won and what it costs per op, how torch.compile captures Python at runtime, what Inductor and Triton actually generate, how the caching allocator and CUDA Graphs attack the two runtime overheads, and the three places where none of it works cleanly yet: dynamic shapes, MoE routing, and top-k.
Part JThe fault line
Why the stack looks like this.
- 01Why eager wonTensorFlow 1 made the right performance bet and lost anyway. JAX made the same bet and won the argument without winning the market — and the gap between those two is the whole story.
- 02What eager mode actually does per opPython, the dispatcher, the autograd tape and the allocator all run before your kernel does. Roughly 8 µs of CPU work to launch 2 µs of GPU work.
- 03What graph mode actually buysThree overheads, three mechanisms, three different ceilings. Knowing which one binds tells you whether compiling will help before you try it.
Part KCapture
From Python bytecode to kernels.
- 04TorchDynamo: capturing Python at runtimeTwo earlier attempts failed by being silently wrong or by demanding a Python subset. Dynamo takes a third route: capture bytecode, guard the assumptions, and bail out when it cannot.
- 05The IR ladder, and the MLIR fork in the roadATen to Prims, functionalization, and AOTAutograd. Then the other lowering universe — MLIR dialects, torch-mlir, StableHLO — and why the CUDA path never goes there.
- 06Inductor: deciding what to fuseThe scheduler is the compiler. Fusion is a graph-partitioning problem under register and occupancy constraints, and the interesting failures are all in where it draws the lines.
- 07Triton: the block programming modelCUDA makes you write per-thread code and manage shared memory. Triton makes you write per-block code and gives the compiler the rest — which is exactly what makes it a codegen target.
Part LRuntime
Memory and launch.
- 08The caching allocatorcudaMalloc synchronises the device, so PyTorch calls it as rarely as possible. The design that follows — pools, splitting, stream ownership — explains almost every confusing OOM you will ever see.
- 09CUDA GraphsRecord the launches once, replay them as one submission. The constraints — static addresses, static shapes, no CPU in the loop — are what the rest of the stack has to bend around.
Part MWhere it breaks
Dynamism the compiler cannot see.
- 10Dynamic shapes and recompilationSpecializing on a shape makes the fastest kernel and the least reusable one. Symbolic shapes are the compromise, and 0/1 specialization is the trap inside it.
- 11MoE: dynamism the compiler cannot removeRouting makes every expert's batch size a function of the data. Capacity factors, grouped GEMM and sort-permute are all ways of buying back a static shape — and each one costs something real.
- 12top-k, sort, and the ops that are not GEMMSelection has low arithmetic intensity, irregular access and a serial dependency. Vocabularies grew to 256k and it became a visible fraction of every decode step.
Part NSynthesis
Choosing, and serving.
- 13Partitioning the stackWhat to compile, what to leave eager, what to hand-write. A decision procedure that starts from which overhead binds, not from which tool is newest.
- 14The inference serverContinuous batching wants shapes that change every step; CUDA Graphs want shapes that never change. Paged KV, bucketing and piecewise compilation are how vLLM and SGLang reconcile them.