Multi-node KV offloading

Pick by scope — how far you need a cached block to be findable — because that is the axis the backends actually differ on.

native — one instance, CPU or shared storage. The right default. No extra dependency, no extra process, no new failure mode, and it captures the biggest win available: a block that fell out of HBM is in DRAM instead of gone. If your traffic has temporal locality within a replica — a user’s conversation returning to the same replica — this is most of the benefit for none of the operational cost. Start here.

LMCache — richer engine, formally supported by llm-d. The distinction worth knowing is that third-party engines “own their own indexing, memory management, and storage” rather than borrowing vLLM’s. That buys smarter eviction, better layout and more tiers; it costs you a second system to understand and operate. llm-d covering it with explicit recipes matters more than it sounds — the difference between a documented path and “follows the same pattern” is measured in weeks.

Mooncake — the one that is genuinely multi-node. Hash-based prefix caching across instances with block-hash deduplication means a block computed by replica A is findable by replica B. That is the capability that changes system design rather than tuning it, and it is the answer to your question about multi-node strategies. It matters most when a shared prefix is used by many users across many replicas — a common system prompt, a shared document, a codebase context — because without it every replica prefills that prefix independently and caches it independently, which is N× the work and N× the memory for one piece of content.

NIXL — not really in this list. It is transport. You use it because you are disaggregating (Module 9), not because you are offloading, and both engines already use it for K3’s prefill→decode handoff. Do not compare it against the others; you may well use it and one of them.

The decision, compactly:

And the thing to check before any of them: whether --enable-prefix-caching is on at all. vLLM leaves it off by default (Module 4). Offloading extends a cache you are not using otherwise.

Because a KV block is not an abstract object — it is a specific memory layout produced by a specific sharding of the model — and a consumer that shards differently cannot interpret it.

The mechanism. Under tensor parallelism, attention heads are split across ranks, so rank $i$ holds the KV for its own heads only. A “block” for token range $[a,b)$ on a TP8 instance contains one eighth of the heads. A TP16 instance expects one sixteenth, arranged differently. Byte-for-byte the cached block is meaningless to it. Hence Mooncake’s stated requirement that all instances share the same tensor_parallel_size — it is not a limitation so much as a statement of what a shared cache can mean.

Which other parallelisms should worry you:

• Pipeline parallelism partitions by layer, so a given rank holds KV for only its layers. A cache entry is therefore per-(rank, layer-range), and two deployments with different PP degrees partition differently. SGLang uses PP8 for prefill on K3 (Module 5), so this is live.
• DCP shards MLA KV by token position. A block cached under DCP8 is one eighth of the positions; under DCP4 it is one quarter. Different DCP degree, incompatible cache.
• Expert parallelism does not affect KV layout — experts hold weights, not cache — so EP degree should be free. This is the one that does not matter.
• Quantized KV is a silent one: a cache written with FP8 KV is not readable by an instance configured for BF16 KV, and nothing about the parallelism configuration tells you that.

How to find out, since the documentation will not tell you:

1. Read the connector’s key derivation. Whatever goes into the block hash defines what must match. If TP rank, layer range or dtype are in the key, mismatched instances simply miss rather than corrupt — which is the safe design and, I would hope, what these implement.
2. Test the negative case explicitly. Bring up two instances with different TP degrees pointed at one store, send the same prefix twice, and check whether the second hits. A cache that never hits is invisible in correctness terms and shows up only as unexplained cost.
3. Monitor hit rate as a first-class metric, not a debug counter. It is the only signal that distinguishes “offloading configured” from “offloading working”, and the failure is silent.
4. Keep the fleet homogeneous. The simplest way to satisfy an undocumented compatibility requirement is not to vary the thing it might depend on. Given how thin the documentation is, I would treat parallelism configuration as part of the cache’s schema and version it accordingly.

It breaks the three properties every offloading system is built on, and the workarounds are the ones Module 2 already established for prefix caching — which is a hint that they are the general answer for hybrid models.

The three broken properties:

• Page granularity. KV offloading moves blocks: you can evict half a sequence’s KV and keep the rest, fetch a prefix without the suffix, and dedupe at block-hash granularity. A KDA state is one object for the whole request. There is no “half a recurrent state”, so it is offloaded whole or not at all, and it cannot be deduplicated against anything.
• Immutability. A KV block, once written, never changes, which is what makes it safe to share across requests and safe to hold in a remote store while a request runs. A KDA state is overwritten every token, so anything you offloaded is stale the moment the request produces another token. Offloading a live request’s KDA state is meaningless; only checkpoints are offloadable.
• Position addressability. A cache lookup asks “do you have the state for this token prefix?” For MLA that is a hash of the tokens. For KDA there is no state for an arbitrary prefix — only for the positions where a checkpoint was taken.

What that forces, and both engines converge on it:

• Only checkpoints are cacheable. vLLM’s interval checkpointing (e.g. every 32 K tokens, plus automatic prompt-end retention) defines the set of KDA states that exist, and therefore the set that could ever be stored or transferred. A prefix that does not land on one replays from the checkpoint below.
• The transfer object is two things. vLLM’s NIXL connector treats a shared page as “token-level MLA cache and request-level KDA state, including convolution and recurrent state”. Any offloading design has to move both, at two granularities, and a partial hit means two lengths to reconcile — which is exactly what vLLM describes comparing.
• Copy-on-write is mandatory, not an optimisation. SGLang lists copy-on-write, snapshot and donate as the operations added to make a mutable state safely shareable. Fetching a checkpoint from a remote store and continuing from it is a branch, and branches copy.

The upside, which is real: because only 24 of 93 layers have per-token KV, the paged portion of K3’s cache is roughly a quarter of what a uniform full-attention model would produce. So there is less to offload per token, and offloading is correspondingly less critical for context length than it would otherwise be. Where it still matters enormously is cross-request prefix reuse — the shared system prompt, the shared document — because that is a capacity-independent win, and it is exactly what Mooncake’s cross-instance store is for.

They solve different problems and the honest answer is that DCP is the stronger lever for context length while offloading is the stronger lever for reuse across requests — so which you need depends on whether your prefixes are shared.

DCP’s case. SGLang’s reported 7.9× logical KV capacity with DCP8 is a large, direct increase in how much context fits, achieved by sharding rather than by moving data anywhere slower. There is no fetch latency, no cache miss, no consistency question: the state is in HBM, just distributed. For “few users, enormous contexts” it is close to strictly better than offloading. Its limits are that it costs an all-to-all per layer, it does not help the KDA state, and — critically — it does not help request count, because per-request overheads are untouched.

Offloading’s case. It attacks something DCP cannot: a block computed for request A being reused by request B, possibly on a different replica, possibly hours later. That is not a capacity multiplier, it is a work eliminator, and llm-d’s “up to 16× faster than recomputing for long prompts” is the relevant figure. For workloads where many users share a large prefix — a system prompt, a document, a repository — this can dominate everything else, because the alternative is every replica prefilling the same 50 K tokens independently.

How to tell which you need, which is a question about your traffic rather than your hardware:

And they compose. Nothing prevents running DCP for capacity and a cross-instance store for reuse; they operate on different axes. The reason to be deliberate is operational cost — a distributed KV store is a stateful system with its own failure modes, capacity planning and consistency behaviour, added to a deployment that already takes 13 minutes to restart.

The measurement that should drive the decision is prefix cache hit rate on your real traffic with native offloading enabled and prefix caching on. If it is already high, a distributed store adds little; if it is low because requests scatter across replicas, that is precisely the gap Mooncake fills, and you will see it as a hit-rate jump rather than as a latency improvement.