The concurrency collapse

Four mechanisms, and only the first is the benign one everybody expects.

1. Ordinary batched-decode sharing, the expected part. In any batched autoregressive server, one decode step produces one token for every sequence, so per-user token rate is (steps per second), and steps get slower as the batch grows. If step time scaled perfectly, 54 concurrent users would each get $16.8 / 1 = 16.8$ tok/s only if step time were unchanged; in reality more work per step means slower steps. A perfectly-amortising dense model would show aggregate throughput scaling nearly linearly, so per-user rate would fall roughly as $1/N$ only once compute saturates. Here aggregate scaled 8.7× for 54× the requests, which is far from linear — so most of the degradation is not this.

2. Expert weight traffic that does not amortise. This is the K3-specific term. A dense model reads its weights once per step regardless of batch. K3 reads the weights of every expert that any token in the batch routed to. At batch 1 that is 16 experts; at batch 54 with random routing it approaches the number of distinct experts touched by 54×16 = 864 draws from 896 — which is most of them. So the weight traffic per step grows substantially with batch while the tokens produced grow linearly, and the ratio worsens. This is why MoE serving has a fundamentally different batching curve from dense serving, and it is the main reason the aggregate scaling is 8.7× rather than something closer to 54×.

3. Load imbalance, which is a tail effect. All ranks run their experts in parallel and the step ends when the slowest finishes. With uneven routing, one rank holds the hot experts and everyone waits for it. Quantile Balancing (Module 1) is designed to attack exactly this, and Moonshot does not publish the resulting imbalance factor — which is the number that would let you predict this term.

4. Capacity-driven queueing, which shows up as latency rather than throughput. p50 latency went from 28.3 s to 60 s and max to 102 s. The spread between p50 and max is the signature of requests waiting for pool space rather than for compute. With a 308,608-token pool, admission is gated, and a request that cannot be admitted contributes latency without contributing throughput.

What this decomposition tells you to do, in order: batch enough to get the aggregate scaling that does exist (the jump from 1 to 54 was still worth 8.7×), but expect the curve to flatten early; measure where it flattens on your context distribution rather than assuming; and treat p50-to-max spread as the signal that you have hit capacity rather than compute, because the fix for those two is different — more GPUs for the first, DCP or offloading for the second.

Because they are limited by different resources, and only one of them is contended in the same way.

TTFT is a prefill measurement, and prefill is compute-bound with high arithmetic intensity: a long prompt processes many tokens in parallel, so weights are read once and used for thousands of positions. It parallelises well, it batches well, and on a warm server with prefix caching it may skip most of the work entirely. Chunked pipeline prefill (Module 5) makes it better still. The 0.37–0.54 s figure is for short prompts on a warm server, which is close to a best case — but the structural point holds: prefill has reuse to exploit and decode does not.

Inter-token latency is a decode measurement, and decode is memory-bound at an arithmetic intensity roughly equal to the batch size (codesign series, Module 12). Every step reads expert weights, crosses the all-to-all, and traverses 93 layers, to produce one token per sequence. There is no reuse to find. Adding users adds expert traffic and synchronisation without adding reuse, so the step time grows and everyone’s tokens slow down together.

The operational consequences are significant and are why the disaggregation of Module 9 exists:

• Your two SLOs are limited by different things, so they should be provisioned separately. TTFT wants prefill capacity and prefix cache hit rate; ITL wants decode capacity and pool headroom. DigitalOcean’s write-up describes exactly this — “staggered TTFT targets by prompt length” with “separate inter-token latency SLAs for chat versus agentic workloads”.
• A user perceives them differently. A 500 ms TTFT with 2.7 tok/s afterwards feels responsive then agonising. Most users would prefer 1.5 s TTFT and 8 tok/s. Since the two are separately tunable — chiefly by choosing your concurrency ceiling — this is a product decision that a systems team is usually making implicitly.
• Prefix caching helps only one of them. It cuts prefill work, so it improves TTFT and total capacity, and does nothing for the decode step rate. On agentic workloads with large shared prefixes it is the single highest-leverage feature — which is why vLLM’s default of leaving it off (Module 4) is worth checking first.

There are three candidate ceilings and the binding one depends on your context distribution — so the answer is a procedure rather than a number.

Ceiling 1: KV/state capacity. Concurrency × context ≤ pool. With the measured 308,608-token pool, that is ~150 requests at 2 K, ~37 at 8 K, ~9 at 32 K. This binds for long-context workloads and is the one DCP attacks: SGLang’s reported 7.9× logical KV with DCP8 moves a 308 K pool to a 2.4 M-class one, which turns ~9 concurrent 32 K requests into ~70.

Ceiling 2: the running-request cap. SGLang states this directly as a limitation — “after DCP lifts MLA capacity, the running-request cap becomes the binding limit.” Once you solve capacity, a scheduler parameter is next, and it exists because per-request overheads (KDA state, metadata, scheduling) do not shrink. This is the ceiling for short-context high-concurrency workloads.

Ceiling 3: acceptable per-user latency. Usually the real one. If your product needs 10 tok/s per user and you measure 2.7 at concurrency 54, your ceiling is wherever the curve crosses 10 — somewhere well below 54 on that hardware. This is a business constraint wearing a systems costume, and it is almost always tighter than the other two.

The procedure:

1. Characterise your context distribution, including the tail. Mean context is not enough because capacity is consumed by the actual lengths, and a few very long requests can occupy the pool.
2. Measure the pool on your configuration. Do not use 308,608 — that is one engine, one hardware, one mem-fraction-static. Get yours from the engine’s startup log.
3. Sweep concurrency and plot both curves: aggregate throughput and per-user token rate. The knee in the first and your SLO crossing on the second bracket the answer.
4. Watch p50-versus-max latency spread. Widening spread means queueing for capacity, which tells you which ceiling you hit.
5. Then decide which ceiling to spend money on. Capacity-bound → DCP, offloading, or more GPUs. Compute-bound → more GPUs or better kernels. Latency-bound → lower your concurrency ceiling and add replicas, which is the expensive answer and often the correct one.

The number I would carry as a prior, pending your own measurement: on a Hopper-class two-node deployment, useful concurrency at moderate context is in the tens, not the hundreds. That is a very different operating point from a 70 B dense model, and plans imported from one will not survive.

It helps at low concurrency, stops helping in the middle, and can hurt at high concurrency — and both engines have published evidence of the transition.

Why it helps at batch 1. The machine is idle. Verification uses compute that would otherwise go unused, and each accepted token skips a full step’s worth of expert reads, all-to-all and 93-layer traversal. Hence 3.14× (vLLM) and ~3.7× (SGLang).

Why the benefit shrinks as the batch grows. Verification is real compute over draft tokens. On a loaded server that compute competes directly with producing tokens for other users. The rejected drafts are pure waste — you did the work and threw it away. As utilisation rises, the opportunity cost of that waste rises with it, and at some point the machine would produce more total tokens by not speculating.

The published evidence. SGLang’s confidence-scheduled verification — which trims the verification window per position using a trained confidence head instead of verifying whole draft blocks uniformly — is reported at +68% throughput at batch 256 with accept length ~2.7, and as “break-even to mildly negative below batch size 8.” Read that carefully: it is a statement that the optimisation to speculation inverts sign with batch size, which strongly implies the underlying trade does too. The feature exists because naive speculation at batch 256 was leaving throughput on the table.

And the workload dependence compounds it. Acceptance is ~4.73 tokens per step on coding and ~2.61 on creative writing (vLLM). Low acceptance means more wasted verification per accepted token, so high-entropy workloads hit the crossover at lower concurrency than low-entropy ones do.

What to do about it:

• Treat speculation as a load-dependent setting, not a global on/off. Enable it aggressively for low-concurrency, latency-sensitive paths and back it off as the batch grows — which is what confidence scheduling automates.
• Route by workload if you can. Coding and tool-calling traffic gets more from speculation than open-ended generation does.
• Never benchmark capacity with speculation on and quote it as throughput. It is the single easiest way to produce a number your production server will not reproduce.