The memory budget

Because the two numbers answer different questions, and only the smaller one is about memory.

8 GPUs is a capacity statement. 1.56 TB of weights across 8×288 GB is ~195 GiB per GPU, which fits with room left over. It is the minimum shape on which the model exists.

64+ is a communication statement. K3 activates 16 of 896 experts per token, and expert parallelism is how you avoid replicating 2.78 T of weights everywhere. The all-to-all that dispatches tokens to expert-holding ranks and gathers the results back is the collective this model lives or dies on, and its behaviour improves with a larger high-bandwidth domain for reasons the codesign series’ Module 11 sets out:

• Experts per rank falls as ranks rise. With EP8 each rank holds 112 experts; with EP64 it holds 14. Fewer experts per rank means less weight traffic per rank per token, which matters because decode is bandwidth-bound.
• Load imbalance averages out. The step ends when the slowest rank finishes. A rank holding 112 experts sees the sum of 112 experts’ loads, but so does everyone else — the variance that hurts is in how hot the hottest rank is, and spreading experts over more ranks with the same top-$k$ reduces the chance any one rank is disproportionately loaded.
• You need the domain to be high-bandwidth, not merely large. vLLM states that multi-node expert/data parallel deployments require RDMA or NVLink. 64 accelerators inside an NVL72-class domain is a very different machine from 64 accelerators across eight Ethernet-connected nodes, and only the first is what the recommendation means.

The honest reading of the gap: 8 GPUs is what you deploy to evaluate the model or serve a handful of internal users. 64+ is what you deploy to serve traffic economically. The published single-user numbers — 111–118 tok/s on vLLM, ~113 on SGLang — come from 8- and 16-GPU configurations, and the aggregate throughput numbers that make the economics work (SGLang’s 2,808 tok/s per GPU under disaggregated PP8 prefill plus TP8 decode) come from larger, disaggregated shapes. Quoting the two together as though one implies the other is the most common error in the coverage.

It is not a contradiction, but it is a much sharper constraint than the spec sheet implies, and the resolution is that the 1 M window is a model capability and the pool is a deployment budget.

What the 1 M means. The model was trained to attend over a million tokens and will produce coherent output at that length. Nothing about that claim promises any particular deployment can hold a million tokens of state.

Why 308 K on that cluster. 16×H200 is a shape chosen to fit the weights, not to leave cache room. After ~102.75 GB of weights per 141 GB GPU, at a static fraction of 0.85, 7.95 GB remains. K3’s hybrid state is much cheaper per token than a uniform full-attention model’s — Module 2’s ~4× argument — but cheaper is not free.

How you actually get to a million tokens, and each of these is a real strategy from the reports:

• More GPUs. The obvious answer: the pool scales with the number of GPUs once the weights are amortised across them. This is the main reason the production recommendation is 64+.
• Decode context parallelism. SGLang’s DCP shards MLA KV by token position across ranks, one $1/N$ per rank, reconciled with a single all-to-all per layer via log-sum-exp reduction. They report 7.9× logical KV capacity on K3 with DCP8 — which converts a 308 K-class pool into a 2.4 M-class one. Note that KDA layers stay tensor-parallel, because a recurrent state cannot be sharded by position.
• Offloading. Push cold KV to CPU DRAM or NVMe and pull it back on demand. Module 8.
• Accept the trade. Serve long contexts at low concurrency, or high concurrency at short contexts, and route the two to different pools.

The constraint to internalise is that context length and concurrency multiply. Your pool holds a fixed number of tokens; a request consumes context-length tokens for its whole lifetime; so concurrent requests × context length ≤ pool. There is no configuration in which you serve many users at a million tokens each on a cluster sized to hold the weights, and DCP is the only technique in the list that changes the constant by an order of magnitude rather than incrementally.

It matters for exactly one thing, and it is worth being clear about which, because it is easy to over- or under-react.

It does not matter for steady-state serving. A 13-minute start amortised over days of uptime is nothing, and the 39 seconds of CUDA graph capture and 25 seconds of distributed init are noise.

It matters enormously for anything that restarts. Concretely:

• Autoscaling is effectively off the table at this granularity. You cannot respond to a traffic spike by bringing up another replica in 13 minutes. Capacity must be provisioned ahead of demand, which changes the cost model in Module 9 — you are paying for peak, not for average.
• Rolling deployments are expensive. Every config change, engine upgrade or model revision costs 13 minutes per replica, and you need spare capacity to cover the replica that is down.
• Crash recovery is a real outage. A rank dying takes the whole tensor-parallel group with it, and recovery is a full reload. At 16-GPU scale across two nodes, the probability of some rank failing over a long run is not negligible, and there is no partial recovery.
• It compounds with the failure modes below. The same report notes that the deployment needed explicit NIC pinning via NCCL_SOCKET_IFNAME and GLOO_SOCKET_IFNAME, and a local Triton JIT cache to avoid filesystem contention between ranks. Each of those was presumably found by a 13-minute cycle ending in a failure.

What can be done, in rough order of effectiveness: keep replicas warm and over-provision rather than autoscale; use a fast shared filesystem or local NVMe for weights, since 1.56 TB at 11.2 minutes implies roughly 2.3 GB/s, which is far below what local NVMe delivers and suggests the load path was the bottleneck rather than anything intrinsic; and separate the prefill and decode pools (Modules 4 and 5) so that a decode replica restarting does not take prefill capacity with it.

The general point is that the cold-start cost is not an inconvenience but an input to the architecture. It pushes you toward fewer, larger, longer-lived replicas — which is the same direction the memory budget and the expert-parallelism requirement push, and it is why K3 deployments look nothing like the elastic fleets people are used to running smaller models on.

Work backwards from the token pool, because that is the binding constraint and the only quantity with a measurement attached.

Step 1 — state the workload in tokens of resident state. Concurrency × mean context length, plus headroom for the tail. 50 concurrent agent sessions at 100 K context is 5 M tokens of state. 200 chat users at 8 K is 1.6 M.

Step 2 — divide by measured per-GPU pool, not by an estimate. The one published figure is 7.95 GB supporting ~308,608 tokens on H200 at TP16/EP16 with mem-fraction-static=0.85. If your hardware and config differ, that number changes, and it changes in ways framework overhead makes hard to predict — so measure it on your shape before planning on it. Note also that the pool is per GPU but the state is sharded, so the aggregate token capacity depends on your parallelism: TP16 replicates KV across ranks in the naive case, while DCP8 shards MLA KV by position for a reported 7.9× logical capacity. Getting this wrong by a factor of 8 is easy.

Step 3 — check the expert-parallel domain separately. Memory sizing may give you a number below what expert parallelism wants. Moonshot’s 64+ recommendation exists because EP behaves better in a large high-bandwidth domain, and vLLM requires RDMA or NVLink for multi-node EP. If your memory calculation says 16 GPUs and they are in two Ethernet-connected nodes, you have satisfied the wrong constraint.

Step 4 — decide prefill and decode separately, then disaggregate. They want different shapes. SGLang reports PP8×TP1 chunked-pipeline prefill delivering 1.7× TEP8’s ceiling, and separately DCP8 for decode capacity; vLLM reports a TEP8 prefill routed to DEP16 decode configuration. If you size one pool for both, one of the two is misprovisioned — which is the codesign series’ Module 12 conclusion arriving as a deployment recipe.

Step 5 — over-provision for cold start. 13 minutes to recover means your steady-state utilisation target has to leave room for a failed replica, because you cannot replace it quickly.

And the honest caveat, which belongs in any plan built on this: every number above comes from day-0 reports published within days of the weights, on hardware ranging from H200 to GB300 NVL72, with different engines and configurations. They are not comparable to each other and they will move. Use them to size a first deployment to measure on, not as a specification.