The inference server

Because you must allocate for the maximum possible length while the actual length is unknown, and the difference is dead memory that cannot be used by anything else.

Pre-paging, each sequence gets a contiguous buffer sized to max_model_len — say 8 K tokens at ~128 KB each, so 1 GB per sequence for Llama-3-8B. A request generating 200 tokens uses 2.5% of it. The rest is reserved, unusable by other sequences because it must stay contiguous for this one, and your achievable concurrency collapses: memory that could hold dozens of real conversations holds a handful of hypothetical maximum-length ones.

PagedAttention replaces the contiguous buffer with fixed-size blocks (16 tokens) allocated on demand from a global pool, plus a block table per sequence mapping logical positions to physical blocks. Waste falls to the last partial block — under 4%.

What that unlocks beyond memory efficiency, and this is the part that matters more:

• The kernel signature stops changing. Attention takes a block-table pointer and a length rather than a variably-shaped tensor, so its arguments are fixed-shape even as the sequence grows — which is what makes CUDA Graph capture possible at all.
• Copy-on-write sharing. Blocks are reference counted, so beam search and parallel samples share the prompt’s blocks instead of copying them, and a shared system prompt is stored once across every request that uses it (prefix caching).
• Preemption becomes cheap. A sequence can be evicted and its blocks reclaimed, then recomputed or swapped back, which gives the scheduler a real admission-control mechanism.

The one-line framing: paging converts a capacity problem into an indirection problem, and GPUs tolerate one level of indirection far better than they tolerate 70% waste.

By separating what actually varies from what merely appears to.

Per decode step, three things could vary: the number of sequences, each sequence’s KV length, and which sequences are present. Handle them differently:

• KV length is removed by paging. The attention kernel reads a block table and a length tensor, both of fixed shape, so growth changes values rather than shapes.
• Identity never mattered — the kernels do not care which requests occupy which rows.
• Batch size is the only genuinely varying dimension, and it is handled by bucketing: capture graphs at 1, 2, 4, 8, … and pad the running batch up to the next bucket. Padding rows compute garbage that is discarded, bounded by the bucket ratio and typically a few percent of real work.

So the decode step becomes: schedule, pad the batch to a bucket, copy the block tables and lengths into the captured input buffers, replay(). One driver call for the whole model.

Prefill is handled separately because it is a different regime — compute bound with long sequences — and is usually run eagerly or with a separate compiled path, since its shapes vary too widely to bucket usefully.

The piece to appreciate is that this is not a workaround so much as a reformulation: the workload looked dynamic because the data structures made it look dynamic. Replace them with structures whose interfaces are fixed-shape, and the dynamism moves into tensor values, where CUDA Graphs are perfectly happy with it. That move — dynamism in values rather than in shapes — is the single most useful idea in this module.

It is compiling and capturing the model in pieces, with the shape-dependent operations left outside the captured regions — and it was forced by attention refusing to fit the same mould as everything else.

The MLPs, projections and normalisations depend on batch size but not on sequence length, so once the batch is bucketed their shapes are fixed and they capture cleanly. Attention is different: even with paging, the kernel’s work depends on the sequence lengths in the batch, and the best implementations (FlashAttention, FlashInfer) are hand-written kernels with their own launch heuristics and sometimes their own host-side dispatch. Forcing them inside a capture means either giving up the good kernel or capturing a worst-case configuration.

So the model is split: capture the dense arithmetic, call attention through a stable interface between the captured regions. vLLM’s implementation compiles the model with Inductor while marking the attention op as an opaque custom op, then captures graphs around it.

The costs are real and worth stating. You get several graph replays per step instead of one, each paying the fixed ~5–10 µs (Module 9). There is host work between pieces, which is exactly what capture was removing. And the split points must be chosen so the boundary tensors have stable addresses.

It is still clearly right, because the alternative is worse in both directions: no capture at all costs milliseconds per step, and capturing a worst-case attention configuration wastes more than the launches saved. This is Module 13’s partitioning rule applied inside a single forward pass — compile what is shape-stable, leave the rest at a well-defined boundary.

They fight because they are different workloads competing for one device, and running either exclusively starves the other.

Prefill processes a whole prompt at once: a large GEMM over thousands of tokens, compute bound, using the machine efficiently. Decode produces one token per sequence: tiny GEMMs, memory-bandwidth bound, launch sensitive, and efficient only through concurrency across sequences.

Run a 4 K-token prefill and every decoding sequence stalls for its duration — hundreds of milliseconds of inter-token latency appearing as a visible stutter to users mid-generation. Prioritise decode and new requests wait, so time-to-first-token climbs and the queue grows.

Chunked prefill splits a long prefill into pieces of a few hundred tokens and schedules each chunk alongside decode work in the same batch. The step now contains a chunk of prefill plus many decode rows, so:

• No decode sequence waits longer than one chunk, bounding inter-token latency.
• The prefill chunk provides the arithmetic intensity that decode lacks, so a memory-bound step gains useful compute — the two workloads are genuinely complementary rather than merely coexisting.
• The batch composition becomes more uniform, which helps bucketing and capture.

The cost is scheduler complexity and a modest increase in total prefill time from splitting a well-shaped GEMM. Everyone accepts it, because latency predictability is the product.

The general statement, and it is the hardware series’ argument arriving intact: this is a queueing problem, and the fix is not making either workload faster but changing the service discipline so a long job cannot block short ones. Head-of-line blocking, solved the way it always is.

Because they can prove things PyTorch cannot, and every proof buys back an optimisation the general runtime had to give up.

PyTorch’s allocator is dynamic because the next allocation is unknown (Module 8). A server knows the model, the maximum context and the memory budget, so it preallocates the entire KV cache as one arena at startup and manages it with a block table — no allocator on the hot path at all.

PyTorch’s dispatcher is general because any op may appear (Module 2). A server runs one model architecture, so the sequence of kernels is fixed and can be captured wholesale.

PyTorch’s scheduler is the Python interpreter. A server has a real request scheduler with admission control, preemption and priorities, because it knows what a request is — a concept PyTorch does not have.

So vLLM and SGLang are not competing with PyTorch; they are specialising it, in exactly the way Module 13 describes. They use PyTorch for the model definition and the kernels, and replace the parts of the runtime whose generality is no longer needed.

The pattern generalises well beyond serving, and it is the note to end the series on: generality costs performance, and the way to recover it is to find the constraints your deployment actually satisfies and specialise against them. Eager mode gave up compile-time knowledge to gain expressiveness (Module 1); torch.compile recovers it where a graph can be captured; a serving framework recovers more because it knows the model and the workload; and a hand-written kernel recovers the rest because it knows the algorithm. Every layer in this series is the same trade made at a different scope.