Latent Diffusion Architecture

How Stable Diffusion-class systems generate: text encoder → denoising loop in latent space → decoder

Latent Diffusion Architecture How Stable Diffusion-class systems generate: text encoder → denoising loop in latent space → decoder Latent space (64×64×4) User Prompt · "a cat on a beach" · Architecture component User Prompt "a cat on a beach" Text Encoder · CLIP / T5, frozen · Architecture component · token embeddings Text Encoder CLIP / T5, frozen token embeddings Gaussian Static · z_T ~ N(0, I) · Architecture component Gaussian Static z_T ~ N(0, I) Sampler · DDIM / DPM-Solver / flow · Latent space (64×64×4) · 20-50 steps Sampler DDIM / DPM-Solver / flow 20-50 steps Denoiser · U-Net / DiT + CFG · Latent space (64×64×4) · noise prediction Denoiser U-Net / DiT + CFG noise prediction VAE Decoder · latent → pixels, once · Architecture component VAE Decoder latent → pixels, once Generated Image · 512×512 px · Architecture component Generated Image 512×512 px Training Images · dataset · Architecture component Training Images dataset VAE Encoder · pixels → latent, ~48× · Architecture component · training only VAE Encoder pixels → latent, ~48× training only tokenize cross-attention start z_t + step t predicted noise clean z_0 decode once compress training latents Legend External Backend Message bus Cloud Frontend

The loop is the cost

  • • Denoiser runs 20-50× per image
  • • Everything iterated lives in latent space
  • • Decoder runs exactly once

Why latents

  • • ~48× smaller than pixels
  • • Attention cost collapses super-linearly
  • • VAE decoder re-synthesizes texture

Steering

  • • Prompt enters via cross-attention
  • • CFG runs the denoiser 2× per step
  • • Encoder absent at generation time

Architecture diagram • Built with Archify • Create yours ↗ • Hover to trace • R route • Click to focus • +/− zoom • M radar • [/] views • P play story • T theme • E export