Tool calling and the API surface

That serving-stack conformance failures cluster, and that a raw failure count tells you almost nothing about how much work you have left.

What happened. DigitalOcean ran Moonshot’s Kimi Vendor Verifier against their deployment and got 30 failures spread across a six-benchmark suite. Read as a scorecard, that looks like a broad quality problem — failures in tool calling, in schema conformance, presumably in agentic coding. Read as a diagnosis, 26 of them were the same missing thing: dynamic tools support. One feature, implemented once, cleared 87% of the failures.

Why it clusters like that. The benchmarks are not independent. Tool calling, JSON-schema accuracy and SWE-Bench all exercise the same API surface, so a single gap in that surface fails all three. This is the norm rather than the exception for serving conformance: the model’s weights are fixed and correct, so almost every failure is in the layer that formats requests and parses responses, and that layer has a small number of features each used by many tests.

The practical consequences:

• Triage by cause before by count. A dashboard showing “30 failures across 6 benchmarks” invites the conclusion that the model or the quantization is wrong. Grouping by root cause first would have shown one missing feature immediately.
• Do not judge a deployment by an early conformance number. 30 failures sounds like a fundamentally broken serving stack; it was a fundamentally working one with one gap.
• Run the vendor’s suite before you build on top. The value of KVV here is not the score, it is that it surfaced a missing feature that would otherwise have appeared as mysterious agent misbehaviour in production. Moonshot publishing a conformance suite at all is the useful thing, and running it is cheap relative to what it catches.

And the observation I would generalise: on a model this well-supported by its engines, the performance work described in Modules 4–9 is largely done for you by vLLM and SGLang, while the API conformance work is not. That is where a deployment’s remaining risk actually lives, and it gets a fraction of the attention that tok/s does — including in the coverage this series is built from, where one paragraph on tool calling sits under many pages of throughput numbers.

Because it changes the failure from “detect an invalid output after the fact” to “make an invalid output unreachable”, and those have very different reliability properties.

How parsing fails. The model generates freely; the parser inspects the result and tries to extract tool calls. If the model emitted a format the parser does not recognise — a variant delimiter, an extra wrapper, a subtly different token sequence — extraction returns nothing. The generation was already produced and paid for; you discover the mismatch afterwards, and the only recovery is to retry and hope.

How constrained decoding works. XGrammar maintains, at each decoding step, the set of tokens that could continue a valid parse of the target grammar, and masks the rest before sampling. Tokens that would produce malformed output have zero probability of being selected. The model cannot emit an unparseable tool call, because the tokens that would do so are unreachable. That converts a runtime detection problem into a construction guarantee, and it is why vLLM integrating K3’s format with XGrammar is the substantive fix rather than a convenience.

What it costs:

• Throughput. Computing the valid-token mask at each step is real work, and it is on the critical path of sampling. Modern implementations are efficient — precompiled grammars, cached mask computation — but it is not free, and it applies only to the structured regions, so the cost scales with how much of your output is constrained.
• Expressiveness. The grammar must actually describe what you want. An over-tight schema forbids outputs the model would legitimately produce; a loose one lets malformed-but-parseable results through.
• It cannot fix semantic errors. Constrained decoding guarantees the tool call is well-formed and matches your schema. It cannot guarantee the model called the right tool with the right arguments. Schema conformance and correctness are different properties, and only one is being bought.

The layered defence that follows, which is vLLM’s own recommendation and which I would treat as the minimum for a production agent:

1. Constrain generation with strict or structured tool calling wherever the schema is known.
2. Validate against your schema anyway on receipt — belt and braces, and it catches version skew between what you constrained and what you expect.
3. Treat empty tool_calls as a distinct outcome, not as “no tool needed”. Retry, or fall back to an unconstrained call, or escalate — but do not let the agent proceed as though the model made a choice.
4. Alert on the rate. Empty-tool-call frequency is a health metric. A step change in it means a model, engine or prompt change has moved the output format, and it is otherwise invisible.

Because the model’s output format is model-specific and the API contract is not, and someone has to own the translation between them — but the design has a real cost that shows up as this class of bug.

The structural situation. K3 emits a single token stream containing reasoning, answer text and tool calls, delimited by model-specific control tokens. The OpenAI-compatible API expects three separate fields. The mapping between them is a property of this model family — the delimiters, the escaping, the ordering — so it cannot live in the generic server code, and it is shipped as a named parser you select with --reasoning-parser kimi_k3 and --tool-call-parser kimi_k3.

Why they are opt-in rather than inferred. In principle the server knows which model it loaded and could pick the parser automatically. In practice: fine-tunes change output formats while keeping the base model’s identifier; some users want the raw stream; and parser selection is decoupled from model loading in the engine’s architecture. The result is a configuration that must be right and is not checked — nothing validates that your parser matches your model, and a mismatch produces plausible garbage rather than an error.

The failure this design produces, and it has now happened twice in this model family:

• Parser omitted. Control tokens appear in the content field as literal text. This is the reported K2 symptom on vLLM 0.10.0 with --enable-auto-tool-choice and --tool-call-parser kimi_k2 — the model “outputs control tokens as text instead of actually invoking functions”. From the outside it looks like a model capability problem, which sends people to investigate the wrong layer entirely.
• Parser present but the model drifts. vLLM’s reported K3 issue: the model occasionally emits a format the parser does not expect, and you get empty tool_calls. Prompt-dependent, so it will not reproduce reliably.

What I would do about it, given the design is what it is: pin the engine version and the parser name together and treat them as one artefact; add a startup smoke test that issues one tool-calling request and asserts a structured result, so a mismatch fails at deploy rather than at 3 a.m.; and monitor the shape of responses, not just their status codes. All three are cheap, and the class of bug they catch is the one that does not announce itself.

That it is the least-hardened part of the stack, and that the model being correct tells you nothing about whether the interface is.

The reasoning behind that, which is a little unfair but I think accurate: the performance work in Modules 4–9 had two well-resourced engine teams, plus Moonshot and NVIDIA, optimising it for weeks before release, with published measurements and independent replication. The API surface had a parser module and a note in a blog post. Kernel work gets benchmarked publicly and compared between engines; tool-call formatting does not. So the maturity gradient runs steeply from the bottom of the stack to the top, and the risk in your deployment sits at the top.

The assumptions I would build on:

• Every tool call may fail to parse. Not often — vLLM describes it as occasional and run-dependent — but often enough that an agent making thousands of calls will see it. Handle it as a fault path with retry and fallback.
• Constrained decoding is the default, not an optimisation. If your schema is known, constrain it. The throughput cost is small relative to the cost of an agent taking a wrong action.
• Validate everything on receipt. Against your schema, not against “is it JSON”.
• Run the vendor conformance suite before launch. KVV exists, covers tool calling and JSON-schema accuracy specifically, and in the one published run it surfaced a missing feature behind 26 of 30 failures. That is a very high return on an afternoon.
• Version-pin the whole surface together. Engine version, parser name, grammar backend, and your schemas. Any of them moving independently can change the output shape.
• Instrument response shape, not just status. Rate of empty tool_calls, rate of schema validation failures, rate of retries. These are the only signals that catch a silent format drift, and none of them are in a default dashboard.

And one thing I would not assume: that this stays true. These reports are from within days of the 27 July 2026 weight release, and API conformance is exactly the kind of thing that gets fixed quickly once real agents hit it. The structural point — that the interface layer is less hardened than the kernels, and that its failures are quiet — will outlast the specific bugs.