11 modules · 4 parts · ~184 min

Serving Kimi K3

Eleven modules on putting a 2.78-trillion-parameter model into production, built from the day-0 reports published by vLLM, SGLang and the people who tried it on hardware they already had. The hybrid KDA/MLA attention that makes a million-token context affordable and makes everything else harder, the memory budget, the two engines, the concurrency collapse, multi-node KV offloading and disaggregation, and the tool-calling surface. Every number is linked to where it came from.

Part SThe model

What it is, and what it costs to hold.

01What K3 actually is2.78T total, 104B active, 896 experts at top-16, 93 layers split 69 KDA to 24 gated MLA, native MXFP4. The numbers that matter for serving, and the one that most write-ups get wrong.
02KDA, AttnRes, and the two-state problem69 of 93 layers keep a fixed-size recurrent state instead of a growing KV cache. That is what makes a million tokens affordable — and it creates the hardest engineering problem in the stack.
03The memory budget1.56 TB of weights before a single token. What that leaves for KV on each cluster shape, why the published minimum is 8 GPUs and the production recommendation is 64, and the one measured pool size everyone should start from.

Part TThe engines

vLLM, SGLang, and choosing.

04vLLM's pathOne hybrid KV manager, DSpark speculative decoding at 3.14×, custom collectives at 1.7–4.5× over NCCL — and three limitations they publish that will bite you on day one.
05SGLang's pathOne unified pool allocated from both ends, chunked pipeline prefill at 1.7× the tensor-parallel ceiling, decode context parallelism for 7.9× logical KV — and 2,808 tok/s per GPU when prefill and decode are pulled apart.
06Choosing, and why the numbers don't comparevLLM says 370 tok/s, SGLang says 423, and a community report on H200 measured 16.8. All three are honest. What actually separates the engines, and the one head-to-head data point that exists.

Part UWhere it falls over

Concurrency, cache, disaggregation.

07The concurrency collapse16.8 tok/s at one request, 146.9 aggregate at fifty-four — 8.7× more total throughput for a 6.2× drop per user. Why long context and many users are the same budget, and what actually sets the ceiling.
08Multi-node KV offloadingFive backends behind one vLLM connector interface — native, LMCache, Mooncake, 3FS, NIXL — three storage tiers, and the constraint nobody documents: which of them work with which parallelism, and what a recurrent state does to all of it.
09Disaggregation and cache-aware routingPrefill and decode want opposite hardware and opposite parallelism, so both engines separate them. What crosses the boundary, why K3 makes the transfer harder than usual, and why the router has to know what is cached where.

Part VOperating it

The API surface, cost, and what to do.

10Tool calling and the API surfaceThe model can be running perfectly and your agents still fail. Reasoning and tool-call parsers, grammar-constrained decoding, the empty-tool_calls failure, and a vendor conformance suite where 26 of 30 failures were one missing feature.
11Cost, break-even, and what to actually doPublished break-even points span 2 billion to 24 billion tokens a month, and the spread is mostly assumptions rather than disagreement. The decision procedure, and an honest account of what these day-0 numbers are worth.