What K3 actually is

Because it changes the two quantities that dominate MoE serving cost — expert weight bandwidth and all-to-all payload — and it changes them by the same factor.

In a conventional MoE, a token routed to an expert is dispatched at full model width. With $H = 7168$ and top-$k = 16$, every token’s dispatch carries $16 \times 7168$ values across the expert-parallel all-to-all, and every active expert’s weights must be read at that width. LatentMoE projects the token to 3,584 first, so both the dispatched activation and the expert’s own weight matrices are half as wide. The all-to-all payload halves; the expert weight traffic halves.

Why that is a serving fact rather than a modelling one:

• Decode is bandwidth-bound, as the codesign series’ Module 12 argues at length. Per-token cost at decode is dominated by reading the active expert weights, so halving that width is close to a direct throughput multiplier on the MoE portion of the step.
• The all-to-all is the hardest collective to overlap, because its sizes are data-dependent (codesign Module 11). Halving its payload halves the exposed time of the one communication you cannot schedule statically.
• It buys routing multiplicity you would otherwise not afford. The design lets K3 use top-16 where the same bandwidth budget would have bought top-8 at full width. That is the actual trade — the latent projection is not free in quality terms, and Moonshot spent the savings on activating more experts rather than on being cheaper.

The consequence for capacity planning is that you cannot estimate K3’s decode cost from total parameters or even from active parameters alone. The 104 B active figure already reflects the latent width, but the traffic also depends on how much of that 104 B is expert weights read at 3,584 versus dense components read at 7,168. Any back-of-envelope that treats the model as a uniform 104 B dense model will be wrong in a direction that depends on batch size.

It moves quantization from a decision you make into a property of the artefact, and that has consequences in both directions.

What it removes. There is no accuracy-versus-memory tier to choose. With a BF16 model you weigh FP8 against INT4 against keeping full precision, run evaluations, and own the result. With K3 the 4-bit weights are what Moonshot trained and evaluated — quantization-aware from the SFT stage onward, with MXFP8 activations — so the published benchmark numbers are the numbers for the weights you are running. That is a genuine simplification, and it removes the most common source of “our self-hosted model is worse than the API” incidents.

What it adds. Your hardware’s format support becomes a hard deployment gate rather than a tuning choice. The engines handle this by routing to different kernels:

On a dequantizing path you still get the memory win — the weights occupy 4 bits in HBM, which is what makes the model fit at all — but the compute is done after upconversion, so the arithmetic throughput advantage of a 4-bit MAC is not available. Per the codesign series’ Module 10, that is exactly the “emulated format” case, and it is the right trade at decode (where you are bandwidth-bound and the memory win is the whole point) and a much weaker one at prefill.

The subtler cost is that you cannot go up. If you find K3’s MXFP4 numerics marginal for some workload, there is no BF16 checkpoint to fall back to — the model was trained into this format. The SGLang RL results are interesting here: they report train/rollout KL sitting flat at a “~2e-3 MXFP4 floor”, which is a candid way of saying the format imposes a noise floor that further training does not cross. For inference that floor is irrelevant; for anyone fine-tuning, it is a real constraint and it is why their recipe uses BF16 LoRA deltas over MXFP4 base weights rather than touching the base.

Three reasons, and only one of them is the latent projection.

The latent width halves the traffic, as above. That is the first-order effect on the all-to-all.

Quantile Balancing attacks the variance directly. The codesign series’ Module 9 makes the case that MoE’s real cost is not the routing but the imbalance — all experts run in parallel and the step ends when the slowest finishes, so a hot expert with 2–10× the average load sets the latency for every device. Deriving expert allocation from router-score quantiles rather than from heuristic auxiliary-loss updates targets the tail of the load distribution rather than its mean, which is the statistic that actually matters. Moonshot does not publish the resulting imbalance factor, which is the number a serving engineer would most want.

More experts at fixed active count is a memory problem, not a compute problem. 896 experts means a very large weight footprint, which is why K3 needs a large cluster. But per token you still touch 16 of them. So the expert count drives capacity and the expert parallel degree, while the top-$k$ drives traffic. Those are separable, and the design deliberately pushed on the one that hardware is better at absorbing — HBM capacity across many GPUs — rather than the one it is not.

Where it is genuinely hard, and both engines say so:

• Expert parallelism becomes mandatory rather than optional. vLLM notes that EP is essential given “896 routed experts, 16 active per token, plus shared experts”, and both engines’ best-throughput configurations are expert-parallel rather than purely tensor-parallel.
• Multi-node EP requires RDMA or NVLink. vLLM states this outright for multi-node expert/data parallel deployments. An all-to-all with data-dependent sizes over commodity Ethernet is the case the codesign series’ Module 11 argues does not work, and here it is as a deployment requirement.
• Small-batch decode does not amortise it. At batch 1 you are paying all-to-all latency to activate 16 experts for a single token. This is a large part of why single-user decode sits around 113–118 tok/s rather than higher, and why speculative decoding matters so much on this model — Module 4.