Conditioning and Classifier-Free Guidance — How Text Steers the Image

Everything so far generates an image — any image the training distribution contains. A product needs the image you asked for. Conditioning is how a prompt gets wired into every denoising step, and classifier-free guidance — a two-line trick discovered almost apologetically — is the single knob that most determines whether the output actually obeys the prompt. The famous "CFG scale" slider in every image UI is this post.

By Pratik Dhanave · November 06, 2026 · 5 min read

#Diffusion Models #Classifier-Free Guidance #Text-to-Image #Generative AI #Image Generation #Prompt Engineering

This is post 7 of Diffusion Models from First Principles. Post 6 built the space the denoiser runs in; this post wires the prompt into the denoiser that runs there. It connects naturally to the Multimodal AI series — text-to-image is multimodality run in reverse, and the text encoders that steer diffusion are exactly the contrastively-trained models that series covers.

Conditioning: the denoiser gets a second input

Unconditional diffusion learns “denoise toward some plausible image.” Conditional diffusion learns “denoise toward a plausible image consistent with this label/prompt/layout” — the network simply receives the condition as an extra input at every step, and training pairs each image with its condition. Nothing in posts 2–5’s math changes; the noise predictor just becomes a function of (noisy latent, step, condition).

For text, the condition is produced by a frozen or jointly-trained text encoder — in Stable Diffusion’s lineage, the text tower of CLIP, a model contrastively pretrained to put images and their captions near each other in a shared embedding space; later systems add or substitute large language-model-style encoders (the T5 family) whose richer token sequences markedly improve spelling, counting, and compositional instructions. The prompt enters the denoiser through cross-attention: at each layer, image positions query the prompt’s token embeddings, so “red hat” can attach to the hat region while “wooden table” attends elsewhere. One encoder choice, one attention mechanism — this is the entire text-image interface, and prompt-following quality tracks the encoder’s language understanding more than any other single factor.

Why plain conditioning isn’t enough

Train that conditional model and sample from it, and the results loosely follow the prompt — too loosely. The model has learned the true conditional distribution, which includes all the images only weakly coupled to their captions (training captions are noisy, often generic alt-text). Sampling the honest distribution reproduces that weak coupling. What products need is to over-weight the prompt — to push samples toward the region where the condition is strongly expressed, even at some cost to diversity.

The first solution, classifier guidance, bolted on a separately-trained classifier that could recognize the condition in noisy images, and added its input-gradient (a score, in post 4’s exact sense — an arrow toward “more consistent with the label”) to the diffusion score at each step. It worked, but demanded training a second model on noisy data for every kind of condition — and a classifier over all possible text prompts is barely coherent.

Classifier-free guidance: the model argues with itself

Classifier-free guidance (CFG) gets the same arrow with no classifier at all. Two moves:

Training: randomly drop the condition (replace the prompt with a null token) a small fraction of the time — in practice around 10% of training examples. One network thereby learns both the conditional and the unconditional denoising task.

Sampling: at each step, run the network twice — once with the prompt, once with the null token — and take the difference of the two predictions. That difference is precisely “what the prompt changes about the denoising direction”: an implicit classifier’s arrow, derived from the generative model itself (Bayes’ rule makes this literal — the conditional-minus-unconditional score is the score of the classifier you never trained). Then extrapolate: take the unconditional prediction and add the difference scaled by a guidance scale greater than one. The final direction over-commits to the prompt by exactly the chosen factor.

The guidance scale is the most consequential user-facing knob in image generation. At scale 1, the honest distribution: diverse, loosely prompt-bound. Around 5–9 (typical defaults, in the convention where 1 means unguided): strong prompt adherence, vivid saturated detail. Push far higher and quality degrades distinctively — burnt colors, waxy over-sharpened textures — because extrapolating a difference vector far off-distribution leaves the manifold the denoiser knows. The knob trades fidelity against diversity, and its cost is concrete: two network evaluations per step, doubling post 5’s bill (production systems mitigate with batching the pair, distilling guidance into the weights — post 10 — or related tricks).

CFG also explains a beloved UI feature almost for free: the negative prompt. Replace the null token’s slot with unwanted content (“blurry, extra fingers, watermark”) and the extrapolation pushes away from it — the same arithmetic, aimed in reverse.

The takeaway: conditioning feeds the prompt — encoded by CLIP-style or T5-style text encoders — into every denoising step via cross-attention; that alone follows prompts only as loosely as training captions did. Classifier-free guidance fixes it by co-training an unconditional mode (random condition dropout) and, at sampling, extrapolating conditional-minus-unconditional predictions by a guidance scale: a classifier’s steering arrow conjured from the generative model itself. One knob governs the fidelity-diversity trade, costs 2× evaluations per step, degrades into burnt saturation when pushed, and gives negative prompts by running the same arithmetic backwards.

Key takeaways

Conditioning = the condition enters the noise predictor at every step; for text, a text encoder (CLIP tower, later T5-class) supplies token embeddings that image positions read through cross-attention — prompt-following tracks encoder quality above all.
The honest conditional distribution is too loosely coupled to prompts (caption noise); products need deliberate over-weighting of the condition.
Classifier guidance added a noisy-data classifier’s input-gradient (a score arrow, post 4) — effective but needs an extra model per condition type.
CFG: drop the condition ~10% of training; at sampling, extrapolate (conditional − unconditional) by the guidance scale — an implicit classifier via Bayes’ rule, no classifier trained. Scale ≈ 1 honest/diverse; ~5–9 faithful/vivid; too high → burnt, waxy off-manifold artifacts. Cost: 2× evaluations per step.
Negative prompts are CFG’s difference vector aimed away from unwanted content — same mechanism, reversed.