Few-step models and the inference budget

Because a solver is constrained to approximate the integral of a given vector field, and the exact trajectory of that field is genuinely curved. Four nodes cannot represent a curve whose curvature is concentrated in the middle band (Module 3) — this is an approximation-theoretic limit, not an engineering one. Raising the order does not help, because high-order methods rely on Taylor expansions that are invalid when steps are large (Module 5).

Distillation escapes by changing the problem. The student is not asked to integrate the teacher’s field accurately; it is asked to reproduce the teacher’s endpoint map. Those are different functions. The endpoint map — noise in, image out — can be smooth and well-approximable even when the path connecting them is not, and a network can represent it directly.

Consistency models make this explicit: the training objective enforces $f(x_t, t) = f(x_{t'}, t')$ for any two points on the same trajectory, so the model defines a one-step map rather than approximating an integral of many steps.

Rectified flow attacks the same problem from the other end: rather than approximating a curved path better, make the path straight, so that one Euler step is exact. Same recognition, opposite move.

The transferable statement: when a numerical method plateaus, question the problem it is solving rather than the method. Here the problem was “integrate this field”, and the better problem was “learn this map”.

Because MSE to the teacher’s output produces a conditional mean, and at one step the conditional distribution is wide.

Concretely: from a given $x_T$, the teacher’s multi-step sampler could land on many plausible images, since the stochastic sampler is genuinely one-to-many and even the deterministic one is sensitive to the trajectory. A student trained under MSE against those targets learns their average — and the average of several plausible sharp images is a blurry image. This is the same failure that makes an L2-trained autoencoder blurry (Module 8) and the same reason regression-based image models were superseded.

An adversarial loss changes the objective from “be close to the target on average” to “be indistinguishable from real”. A discriminator penalises blur specifically, because blur is the most easily detectable deviation from the natural image manifold. So the student is pushed to commit to one sharp mode rather than hedging across several.

The costs are the usual GAN costs and they are real: training instability, mode-seeking behaviour that reduces diversity, and a discriminator to tune. Which is why the strongest few-step results combine losses — distillation for the trajectory, adversarial for sharpness, and often a perceptual term — rather than relying on any one.

The general principle: MSE is correct only when the target is unimodal given the input. At many steps the per-step conditional is nearly unimodal and MSE is fine, which is why ordinary diffusion training works at all. Compress to one step and it stops being true.

Start from the identity: $\text{time} \approx \text{NFE} \times t_{\text{fwd}} + t_{\text{decode}} + t_{\text{encode}}$. Two independent factors sit inside NFE, and people conflate them.

Take SDXL at 1024², 20 steps, guided: NFE = 40, roughly 4.5 s.

The first row is the point worth internalising: removing guidance is a free 2× and it is independent of everything else. Guidance distillation is often easier than step distillation and is frequently the larger single win, yet it gets less attention because “20 steps” sounds like the number to attack.

Below about 4 NFE the fixed costs start to dominate — the VAE decode at 80–150 ms is a quarter of a 4-step generation, and at 1 NFE it is comparable to the entire diffusion cost. So the ordering of optimisations changes as you speed up:

1. At 40 NFE: attack NFE. Everything else is noise.
2. At 4 NFE: the decoder and text encoder are now material; a faster decoder (TAESD or similar) matters.
3. At 1 NFE: you are memory-bandwidth-bound on weights, and the levers are quantisation and a smaller model — the batch-1 regime from the other series.

Decide by constraint rather than by benchmark, because the methods dominate in different regimes.

Do you control training?

• No → best solver plus tuned step placement (Modules 3, 5), and LCM-LoRA, which is the one few-step method that composes with existing weights and can be applied to a model you did not train.
• Yes → distillation is available, and the question becomes which kind.

What is the quality bar relative to the teacher?

• Must match → do not distil. Use 20 NFE with a good solver and distil only the guidance, which is a 2× win at essentially no quality cost.
• Small loss acceptable → progressive or consistency distillation to 4–8 steps.
• Interactive latency is the product → adversarial distillation to 1–4 steps, accepting reduced diversity, which for interactive use is often unnoticeable since the user is re-rolling anyway.

Building from scratch? → use a flow-matching objective. Straight trajectories make few-step sampling a property of training rather than a later distillation stage, and you avoid maintaining a teacher-student pipeline at all.

The mistake to avoid is optimising NFE while ignoring that guidance doubles it, and ignoring that below 4 NFE the fixed costs you never measured become the budget.