Solvers: why twenty steps is enough

Because order counts the rate at which error shrinks, not its constant, and the constant is where the win is.

The diffusion ODE in log-SNR coordinates has the form $\frac{dx}{d\lambda} = a(\lambda)\,x + b(\lambda)\,\hat\varepsilon_\theta(x,\lambda)$. The first term is linear in $x$ with a known coefficient, so it can be integrated exactly with an integrating factor — no approximation, no error contribution at all. Only $\hat\varepsilon_\theta$ needs numerical treatment.

A generic Runge–Kutta method sees one opaque right-hand side. It spends its approximation budget on the combined field, including the linear part it did not need to approximate, and its error constant includes contributions from a term that had a closed form. Being second order does not save you from a large constant.

The measurable consequence: DPM-Solver at order 2 substantially outperforms a generic order-2 method at the same NFE, and the gap widens as NFE falls, because at large steps the constant dominates the asymptotic rate.

The transferable lesson is a general one about numerical work: exploit known structure before raising order. Order is the last resort, after you have removed everything you can solve exactly.

Multistep, in this setting, and the reason is the unusual cost asymmetry.

A single-step second-order method (Heun) evaluates the field at the current point, takes a trial step, evaluates again at the trial point, and averages. Two NFE per step. A multistep method (DPM-Solver++(2M)) estimates the derivative from the current and previous evaluations, giving second order at one NFE per step. At equal NFE, multistep takes twice as many steps, so its step size is half, so its error is roughly four times smaller for the same cost.

Why it is not obvious: in classical ODE work, multistep methods carry real disadvantages. They need a startup procedure, they have smaller stability regions, and they behave badly when the step size changes abruptly or the field is non-smooth. Those objections are what make Runge–Kutta the default in general-purpose solvers.

They mostly do not bite here. The field is smooth in $\lambda$, the step schedule is fixed in advance rather than adapted, the integration is short (tens of steps, so error accumulation over long horizons is not a concern), and startup is a single first-order step whose error is subsequently swamped. Meanwhile the function evaluation is a full U-net or transformer forward pass — vastly more expensive than anything else in the loop — so halving NFE per step dominates every other consideration.

The general point: solver selection depends on the cost model, not only on the mathematics. When function evaluation dominates by orders of magnitude, methods that reuse past evaluations win.

You have left the regime where the solver’s error analysis applies.

What is happening. Order-$p$ convergence is asymptotic as $h \to 0$; it assumes the field is well approximated by a Taylor expansion over one step. At 8 steps each step spans a large interval of log-SNR, and in the middle band the trajectory has real curvature (Module 3), so the expansion is simply invalid. Worse, high-order methods can amplify this — they extrapolate using derivative estimates that are themselves wrong, so a third-order solver at 6 steps can underperform a second-order one.

What to do, in order:

1. Fix step placement first. Karras spacing ($\rho=7$) or uniform-in-$\lambda$ rather than uniform in $t$. At low NFE, placement matters more than order — you are choosing where your handful of nodes go.
2. Drop to second order. Not up. DPM-Solver++(2M) is the practical floor.
3. Reduce the guidance scale. High CFG (Module 6) sharpens the field and makes it harder to integrate; part of what looks like solver failure at low NFE is guidance-induced stiffness.
4. Below ~10 NFE, stop solving and start distilling. No solver reaches 4 steps on a model trained for 1000. Consistency models, progressive distillation and adversarial fine-tuning (Module 9) change the model so a coarse trajectory is correct, rather than trying to integrate the original one coarsely.

The line worth internalising: solvers buy you 1000 → 20; distillation buys you 20 → 4. They are different mechanisms and the second cannot be reached by improving the first.

Split it into three sources, because each has a different remedy and conflating them wastes effort.

1. Discretisation error. Reducible by more NFE, better placement, higher order. Measure it by comparing against a 200-NFE reference with the same seed and solver family: the difference is discretisation error, and it is usually concentrated in the middle band where curvature is highest.
2. Score error. The network is not the true score, particularly at noise levels where it got little training weight (Module 2). This is irreducible by any sampler, and it is what the 200-NFE reference still gets wrong. Localise it by evaluating denoising MSE per $\lambda$ against held-out data.
3. Guidance-induced distribution shift. With CFG the field you integrate is not any model’s score, so the sample is not from the model distribution at all (Module 6). This shows up as over-saturation and reduced diversity, and it is not error in the numerical sense — it is a deliberate change of target that people forget they made.

The diagnostic sequence: high-NFE reference to separate (1) from (2)+(3); then set guidance to 1 to separate (3) from (2). Two experiments, and they partition the error completely. Most teams jump straight to changing the solver, which only ever addresses the first of the three.