Disaggregation and cache-aware routing

When the efficiency gain from letting each phase use its preferred parallelism exceeds the transfer cost — which on K3 is a lower bar than usual, because the parallelism gains are unusually large.

The cost side. Every request now moves its MLA KV and its 69 layers of KDA state from a prefill worker to a decode worker before generation starts. That is bytes on the wire, added directly to TTFT, plus a scheduling handoff and a second admission decision. On a short prompt it may dominate; on a long one it is amortised over a large prefill.

The gain side, which is where K3 is unusual. The published deltas are not marginal. Prefill in PP8×TP1 rather than TEP8 is 1.7×. Decode with DCP8 gets 7.9× logical KV capacity, which translates into far higher achievable concurrency and therefore far better aggregate throughput. You cannot have both in one instance. So the comparison is not “unified versus disaggregated with a transfer tax” — it is “one phase running well and the other compromised, versus both running well minus a transfer”.

The conditions that favour disaggregation:

• Long prompts, because the transfer amortises over a large prefill and because prefill’s parallelism advantage grows with sequence length.
• A prefill/decode ratio that is not 1:1. Agentic workloads with big contexts and short outputs need much more prefill capacity per unit of decode; chat with short prompts and long answers needs the reverse. Disaggregation lets you scale the two independently, which is arguably a bigger operational win than the throughput number.
• A fast interconnect between the pools. The transfer is on the critical path, so it wants RDMA or NVLink — the same requirement vLLM states for multi-node EP/DP.

The conditions that favour staying unified:

• Short prompts with long generations, where the transfer is a large fraction of a small prefill.
• Small deployments. Two pools means two sets of replicas, each needing enough capacity to hold 1.56 TB of weights (Module 3). At 8–16 GPUs total you cannot meaningfully split.
• Operational simplicity, which is not nothing when a cold start is 13 minutes and you now have two failure domains, a transfer path, and a router that must know about both.

My reading of the evidence: the aggregate figure both engines highlight is a disaggregated one, and neither publishes a comparable unified aggregate number — which suggests disaggregation is where the throughput is, but also that we are not being shown the controlled comparison. Treat “disaggregate” as the direction of travel at scale and as unnecessary complexity below roughly a few dozen GPUs.

Because it is a different kind of object, and every property that makes a KV transfer easy is absent.

A KV transfer is easy for specific reasons. The pages are immutable once written, so the source can keep serving while the transfer runs and there is no consistency question. They are position-indexed, so a partial transfer is meaningful and can be pipelined — start sending page 0 while page 500 is still being computed. They are uniform, so the transfer is a bulk contiguous copy, which is what NIXL and RDMA are good at. And if something goes wrong, recomputing a page is cheap and local.

A KDA state has none of that. It is one object per request per layer, mutable, with no internal position structure. Specifically:

• It must arrive complete and exact. The decode worker will continue updating it in place, and a state that is subtly wrong produces subtly wrong output forever after — with no error, no divergence signal, and nothing that a health check would catch. This is the same correctness requirement that made ReplaySSM’s bit-identical guarantee necessary (Module 5).
• It cannot be pipelined against prefill. There is no partial KDA state to send early; the state for token $n$ only exists once token $n$ has been processed. So the KDA portion of the transfer begins only when prefill finishes, whereas MLA pages can stream as they are produced.
• It is 69 layers’ worth, plus convolution state. vLLM is explicit that the request-level view includes “convolution and recurrent state” — there is more than one tensor per layer to move.
• You cannot fall back to recompute cheaply. Recomputing a lost KV page means redoing one position; recomputing a lost KDA state means replaying from the last checkpoint, potentially tens of thousands of tokens (Module 2).

What follows for a deployment: the prefill→decode handoff on K3 is a stricter operation than on a conventional model, it has a serial component that cannot overlap prefill, and its failure mode is silent corruption rather than a retry. That argues for keeping the two pools close — same fabric, low latency, RDMA — and for treating the transfer path as a correctness-critical component rather than a performance one. It is also a good example of the pattern running through this whole series: the hybrid architecture that made 1M context affordable moved cost into the serving stack in a form that did not exist before.

Because with $N$ replicas and no routing intelligence, your effective cache hit rate falls by roughly a factor of $N$, and the failure is completely silent.

The arithmetic. A shared prefix — a system prompt, a document, a repository context — is cached on whichever replica first computed it. A new request carrying that prefix arrives and the load balancer picks a replica by round-robin or least-connections. The probability it picks the one holding the cache is $1/N$. With 8 replicas, seven out of eight requests prefill a prefix that already exists elsewhere in your cluster. You pay 8× the prefill compute and hold 8 copies of the same blocks, and every metric you normally watch — error rate, throughput, GPU utilisation — looks fine. Utilisation looks great, in fact, because you are doing a great deal of unnecessary work.

What the fix does. llm-d’s Router uses an Exact Prefix Predictor to select replicas that can reuse cached blocks — routing by content rather than by load. Two things make this more than a hash ring:

• It must trade cache affinity against load. Always sending a popular prefix to the replica holding it creates a hotspot. The router needs to know both what is cached where and how loaded each replica is, and pick a point on that trade.
• It composes with a shared store. With Mooncake’s cross-instance cache (Module 8), a miss on the local replica can still hit the distributed store, so routing becomes an optimisation over transfer cost rather than a hard requirement. The two mechanisms substitute for each other partially: perfect routing reduces the need for a shared store, and a fast shared store reduces the need for perfect routing.

And the K3-specific wrinkle. Because prefix hits on this model must establish both reusable MLA pages and a valid KDA checkpoint (Module 2), a “hit” is a pair of lengths rather than one number. vLLM describes reconciling this: when a local GPU hit conflicts with a remote cache result, the scheduler compares the exact reusable token lengths from both tiers and selects the longer prefix. So the router’s model of what is cached where is more complicated than a prefix hash, and a checkpoint interval that does not align with your workload’s shared prefixes reduces the value of routing as well as the value of caching.

The operational advice, which is short: instrument prefix cache hit rate per replica, and compare it against what a perfect router would achieve given your traffic’s prefix distribution. That gap is the value of cache-aware routing, expressed in the only units that matter. Almost nobody measures it, and on multi-replica deployments with shared prefixes it is frequently the largest single inefficiency in the system.

In four stages, each one justified by a measurement from the previous — because every stage adds a failure domain and a 13-minute restart to something.

Stage 1 — one pool, prefix caching on. Single unified deployment, --enable-prefix-caching explicitly on (Module 4), the parallelism that fits your hardware. Measure: the token pool from the startup log, aggregate throughput and per-user rate across a concurrency sweep, TTFT and ITL separately, and prefix cache hit rate. This is your baseline and most deployments should stop here until something in it is unacceptable.

Stage 2 — add capacity where you are bound. From Module 7’s three ceilings: if capacity-bound at long context, enable DCP; if bound by cross-request recompute with a low hit rate, add native CPU offloading. Both are single-system changes with no new distributed component. Re-measure the same things.

Stage 3 — disaggregate prefill and decode. Justified when your prefill/decode ratio is lopsided enough that one pool is idling while the other saturates, or when you want PP8-class prefill and DCP8-class decode capacity simultaneously. Requires a fast fabric between pools, a NIXL or Mooncake transfer path, and acceptance that the transfer is correctness-critical. Measure the transfer cost as a component of TTFT explicitly, because that is the thing you are paying.

Stage 4 — multi-replica with cache-aware routing and a shared store. Justified when you have enough replicas that the $1/N$ hit-rate problem is real, and enough shared prefix traffic for it to matter. Adds a router with prefix awareness and, if the store is cross-instance, a distributed stateful system with its own operations. Measure hit rate per replica against the achievable ceiling.

What I would resist at every stage: adding a component because the reports mention it. Every number in this series comes from a day-0 post on hardware that is probably not yours, and the techniques described — DCP, disaggregation, distributed KV — each solve a specific bottleneck that you may not have. The sequence above is designed so that each stage produces the measurement that tells you whether the next one is warranted, and the most common good outcome is stopping at stage 2.