SGLang's path

Because the right split between recurrent state and paged KV depends on the workload mix, the mix changes minute to minute, and a fixed split cannot follow it.

What the split depends on. KDA state is a per-request constant; MLA KV is per-token. So the ratio between them is set by mean context length. A hundred concurrent 2 K-token chats is dominated by the 100 fixed KDA states. Two concurrent 500 K-token documents is dominated by MLA KV. Both are K3 workloads, and a production server sees both, sometimes in the same minute.

What two fixed pools cost you. You choose a boundary at startup. When traffic skews short, the MLA region sits idle while the KDA region is full and you reject requests. When traffic skews long, the reverse. You cannot rebalance without a restart, and a restart is 13 minutes (Module 3). SGLang names exactly this: the unified pool eliminates “fragmentation and pre-sizing guesses”.

Why allocating from opposite ends is the right primitive. It is the classic heap/stack arrangement, and it works here for the same reason: two allocators with different lifetime and size characteristics share one contiguous region, each growing toward the other, with the only failure condition being that they meet. There is no internal boundary to get wrong, no unused reservation on either side, and the memory is fully fungible right up to exhaustion. It also suits the objects: KDA blocks are fixed-size and per-request; MLA pages are uniform and numerous. Growing the numerous uniform thing from one end and the chunky per-request thing from the other keeps each side’s allocation pattern simple.

What it does not solve. The pool still runs out, and now it runs out as a single event rather than as two independently diagnosable ones — a long-context request and a burst of short ones now compete directly, so admission control has to arbitrate between them explicitly rather than falling out of separate budgets. That is a policy problem moved into the scheduler, which is the right place for it but is not nothing. It is also why SGLang’s stated limitation after DCP is that “the running-request cap becomes the binding limit”: once you fix capacity, the next constraint is a scheduler parameter.

By sharding the KV along the token axis instead of replicating it, and reconciling the partial attention results with a mathematically exact reduction.

The mechanism. Under plain tensor parallelism, attention heads are split across ranks but every rank needs the KV for the positions its heads attend to — in the common arrangement the KV is effectively replicated or partly replicated across the TP group, so aggregate KV capacity does not scale with rank count. DCP instead gives rank $i$ tokens $i, i+N, i+2N, \dots$ (or a contiguous block), so each rank holds $1/N$ of the sequence. Each rank computes attention against only its shard, producing a partial result: an unnormalised weighted sum plus a running log-sum-exp of the scores. One all-to-all per layer combines them — and because softmax normalisation can be deferred and recombined via log-sum-exp, the combination is exact rather than approximate. It is the same algebraic move that makes FlashAttention’s tiling legal, applied across ranks instead of across tiles.

Why 7.9× and not 8×. With DCP8 you would naively expect 8×. The shortfall is the KDA state, which does not shard, plus per-rank metadata and alignment overhead. That the number is 7.9 rather than, say, 5 tells you the KDA state is a small fraction of the pool at the context lengths they measured — consistent with long-context agent workloads, and it would look worse at short context.

Why KDA cannot participate. A recurrent state is not indexed by token position. It is a single matrix summarising all tokens seen so far, produced by a sequential in-place update. There is no subset of it corresponding to tokens $i, i+N, \dots$ to give to rank $i$, and no reduction that recombines per-rank partial states, because the update is not associative in the way attention’s softmax-weighted sum is. So KDA layers stay tensor-parallel, sharded by feature dimension like any other matmul.

The consequence, which is a real asymmetry: DCP scales your context length but not your request count, because the per-request KDA constant is untouched. So DCP is the right tool for “few users, enormous contexts” — which is exactly the agentic workload they benchmarked, 48 sessions at 100 K context — and does nothing for “many users, short contexts”. Module 7 is about what does.

Because prefill is compute-bound with long sequences, which is the regime where pipeline parallelism’s weakness disappears and tensor parallelism’s cost does not.

Why TP is expensive for prefill here. Tensor parallelism all-reduces activations twice per layer, with bytes proportional to (batch × sequence × hidden). At prefill, sequence is enormous — that is the whole point — so those all-reduces are large and frequent, and there are 93 layers of them. The codesign series’ Module 11 gives the ratio: TP’s FLOP-per-byte is roughly $H$, which is fine on NVLink, but “fine” still means a substantial synchronous cost 186 times per forward pass.

Why PP works well for prefill specifically. Pipeline parallelism’s usual problem is the bubble, $(p-1)/m$ for $p$ stages and $m$ microbatches — you need many microbatches to fill it. Prefill naturally provides them: chunk a long prompt and each chunk is a microbatch. A 200 K-token prompt at reasonable chunk size gives you dozens of chunks streaming through 8 stages of 12 layers, so the bubble is small. And the communication is point-to-point activations at stage boundaries — 7 handoffs instead of 186 all-reduces — which is why they report 91% of communication time hidden and 1.7× TEP8’s ceiling.

What it means for the deployment, and this is the practical part:

• Prefill and decode want different parallelism, so you must disaggregate to have both. PP8×TP1 for prefill and DCP8 or TP8 for decode cannot coexist in one engine instance. This is why the aggregate number (2,808 tok/s per GPU) is quoted for a disaggregated configuration, and vLLM’s equivalent is TEP8 prefill routed to DEP16 decode. Module 9.
• It changes your failure domains. A prefill pipeline stage failing is different from a decode rank failing, and you now operate two pools with different scaling characteristics.
• The transfer between them is now on the critical path. Every request’s KV and KDA state must move from a prefill worker to a decode worker, which is what the NIXL connector’s two logical views exist for. That transfer is pure overhead added to TTFT, and it only pays because the two phases are so much more efficient in their preferred shapes.

Because it is per draft step per sequence in the batch, and it is bandwidth on the exact resource decode is already limited by.

The arithmetic. 512 KB per draft step. At batch 64 with a draft depth of 4, that is $512\,\text{KB} \times 64 \times 4 = 128$ MB of state written and read per decode step. At even 20 steps per second that is 2.5 GB/s of pure bookkeeping traffic — and decode is memory-bandwidth-bound, so every byte spent on snapshots is a byte not spent reading expert weights. The snapshot competes directly with the thing producing tokens.

Worse, it also consumes capacity: those snapshots live in the same pool as the KV and the KDA states (Module 3’s 7.95 GB on H200). Speculation would be reducing your maximum concurrency in order to increase your per-user speed, which is close to the opposite of what a loaded server wants.

What ReplaySSM does instead. Store the raw inputs to the state update rather than the state itself — ~16 KB, a 32× reduction. The verification kernel additionally records the gate values, and a fold kernel replays the accepted prefix to reconstruct the state. It is store-versus-recompute (codesign Module 7) applied to speculation: the inputs are much smaller than the state, and replaying a handful of tokens is cheap compared with the bandwidth of moving the state.

The property that makes it safe, and it deserves emphasis: reconstruction is bit-identical. This is not a nicety. Speculative decoding’s guarantee is that its output distribution matches non-speculative decoding exactly; if the reconstructed KDA state differed even slightly from what sequential processing would have produced, outputs would drift from the non-speculative result in a way that is invisible in aggregate metrics and shows up as unreproducible behaviour. Getting a recomputation to be bit-identical through a gated recurrent update is genuinely fiddly, which is presumably why the gate values are recorded rather than recomputed.

The generalisable point: every technique that assumed append-only KV needed re-deriving for a recurrent state, and the re-derivation was usually “store less, recompute more, and prove the recomputation is exact”. That is the same answer three times — prefix caching (checkpoint and replay), speculation (ReplaySSM), and offloading (Module 8) — which suggests it is the right general shape for hybrid models rather than three coincidences.