Conditioning and control

Because cross-attention binds attributes to regions softly and by learned association, and nothing in the architecture enforces that “red” attaches to the cube’s pixels rather than the sphere’s.

Mechanically: each spatial position produces a query and attends over all 77 text tokens. A position inside the sphere can and often does attend to “red”, because the attention is a similarity in a shared embedding space with no constraint tying colour tokens to the noun they modify. The text encoder itself contributes — CLIP’s pooled representations are famously weak at compositional structure, so “red” and “blue” are not cleanly separated in the conditioning sequence before attention even begins.

What actually helps, in order of effectiveness:

1. A better text encoder. T5 in SD3 and Flux improves compositionality substantially, because the conditioning sequence itself carries the binding.
2. Spatially indexed conditioning. Give each object a region — a layout map, a segmentation mask, regional prompting — so the mechanism is spatial rather than associative.
3. Attention manipulation at inference. Attend-and-excite and similar methods directly optimise the latent so each noun token achieves sufficient attention mass. A patch on the symptom, but an effective one.
4. More guidance. Usually makes it worse, by amplifying whichever binding the model already preferred.

The framing: this is not a capacity problem and more training will not fix it. It is a mechanism mismatch, and the fix is a mechanism with the right inductive bias.

Two separate design decisions, each solving a specific failure.

Freezing protects the base model from catastrophic forgetting on a small conditioning dataset. With 10k–100k pairs, fine-tuning the whole model would degrade its general prior toward that dataset’s distribution. Freezing means the worst case is that the control signal is ignored — the base model’s quality is a floor rather than something at risk. It also means one base model can host many independent ControlNets.

Copying the encoder rather than using a small fresh adapter is about initialisation and feature compatibility. The injected features must live in the same representational space as the frozen backbone’s activations, at every resolution. A copy of the trained encoder already produces exactly such features, so training starts from a good place instead of learning the space from scratch. The zero-initialised convolutions between copy and backbone complete the design: at step zero the injection is exactly zero, so the model is bit-identical to the frozen original and training begins from a known-good state rather than from a perturbation of it.

The cost is real — roughly 35–50% more parameters and 30–45% more latency, since the copied encoder runs every step. Which is why lighter alternatives exist (T2I-Adapter, ControlNet-Lite, and LoRA-style control), trading some fidelity for much less compute. The choice is the usual one: ControlNet when the spatial constraint must be respected precisely, adapters when it is a hint.

Ask two questions, in this order.

Is the signal spatial? If it varies across the image — depth, edges, pose, a mask — it must enter through a spatially indexed path: channel concatenation if it is cheap and aligned with the input resolution, ControlNet or an adapter if it needs to influence deeper structure. Trying to push a spatial signal through a global embedding or a short token sequence discards its spatial content at the first layer, and no amount of training recovers it.

How much data do you have? This decides how much you may perturb the model.

Then the practical checklist: add independent dropout for the new signal so you can guide on it; decide whether it should be composable with existing conditions (which favours additive mechanisms over ones that rewrite shared features); and check inference cost, since anything running every step multiplies by NFE and again by 2 if guided.

The mistake worth avoiding is reaching for the most powerful mechanism by default. ControlNet on a global attribute is a large waste, and a LoRA asked to enforce exact geometry will simply fail.

Because attention maps are where the layout lives, separately from where the content lives — and editing usually means changing one while holding the other.

The observation behind prompt-to-prompt is that in a text-to-image U-net, cross-attention maps at each step encode the spatial arrangement: which regions belong to which token. Value features encode appearance. So if you generate with prompt A, cache its attention maps, then generate with prompt B while substituting A’s maps, you get B’s content in A’s composition. Swap the maps for a token you changed and keep the rest, and you have a targeted edit.

Why this works with no training: you are not modifying the model at all, only reusing intermediate state across two runs. The model’s own learned association is doing the work; you are just holding part of it fixed. It costs nothing beyond the second generation.

Where it breaks:

• Structural changes. If prompt B implies a different layout, forcing A’s maps produces incoherence — you are demanding a composition the content cannot occupy.
• Real images. Requires inversion first, inheriting all of Module 4’s problems, especially under guidance.
• Architectures without cross-attention, or with fused attention where maps are not materialised. FlashAttention does not expose the map, so the technique needs an explicit, slower path.

The generalisable insight is worth more than the technique: intermediate activations of a generative model are an interface. Anywhere the model has factored a representation — layout versus appearance here — you can intervene without training, provided you can identify where the factorisation lives.