How frontier LLMs are actually built — the shift from dense to sparse (compute-per-token welded to parameter count in dense models; conditional computation decouples capacity from per-token compute), the transformer backbone recapped (attention mixes information across tokens, the FFN block MoE replaces), Mixture of Experts core idea (many expert FFNs + a router picking top-k per token; total vs active parameters), routing and load balancing (expert collapse, the auxiliary load-balancing loss, expert capacity and token dropping, expert-choice routing), training and serving MoE (the compute-saved-memory-not trade, expert parallelism, all-to-all communication), the long-context problem (attention's quadratic compute vs the KV cache's linear memory), efficient attention (GQA/MQA shrink the KV cache, FlashAttention for fast exact attention, sliding-window/sparse to break the quadratic), and the anatomy of a modern frontier LLM plus where architecture is heading. Grounded in the MoE/Switch/Mixtral papers, Attention Is All You Need, FlashAttention, and GQA.
The frontier language models of the last few years share a structural secret that isn't obvious from the outside: most of them aren't dense. They're sparse — built from Mixture-of-Experts layers that let a model have hundreds of billions of parameters while only using a fraction of them on any given token. Understanding why models moved from dense to sparse is the key to understanding how modern LLMs are actually built.
Frontier language models share a structural secret that isn't obvious from the outside: most aren't dense. They're sparse — built from Mixture-of-Experts layers that let a model have hundreds of billions of parameters while using only a fraction on any given token. Understanding why models moved from dense to sparse is the key to understanding how modern LLMs are built.
Mixture of Experts is a modification to the transformer, so you can't understand modern LLM architecture without a clear picture of the transformer itself. This post is a focused recap of the parts that matter for the rest of the series: attention as the mechanism that mixes information across tokens, the feed-forward block that MoE replaces, and how the pieces stack into the model everyone is now modifying.
Mixture of Experts is a modification to the transformer, so you can't understand modern LLM architecture without a clear picture of the transformer itself. This post is a focused recap of the parts that matter: attention as the mechanism that mixes information across tokens, the feed-forward block that MoE replaces, and how the pieces stack into the model everyone is now modifying.
Now we open up the mechanism at the heart of modern LLMs. A Mixture-of-Experts layer replaces the transformer's single feed-forward block with many parallel "experts" and a router that sends each token to just a few of them. The result is a model with enormous total capacity that spends only a little compute on any given token — the decoupling the first post promised, made concrete.
A Mixture-of-Experts layer replaces the transformer's single feed-forward block with many parallel experts and a router that sends each token to just a few of them. The result is a model with enormous total capacity that spends only a little compute on any given token — the decoupling of capacity from per-token compute, made concrete. Experts, routing, and how outputs combine.
The router is where Mixture of Experts succeeds or fails. Left to its own devices, it tends to collapse — sending most tokens to a handful of favorite experts while the rest starve, wasting the model's capacity. Making routing spread load evenly, without hurting quality, is the central engineering challenge of MoE, and the techniques for it are what separate a working sparse model from a broken one.
The router is where Mixture of Experts succeeds or fails. Left alone it tends to collapse — sending most tokens to a handful of favorite experts while the rest starve, wasting the model's capacity. Making routing spread load evenly, without hurting quality, is the central engineering challenge of MoE: load-balancing losses, expert capacity, and token dropping.
Mixture of Experts saves compute but not memory — and that single fact reshapes everything about how these models are trained and served. All the experts must exist in memory even though only a few run per token, which turns MoE into a distributed-systems problem as much as a machine-learning one. This post covers the trade MoE actually makes and the parallelism it forces.
Mixture of Experts saves compute but not memory — and that single fact reshapes everything about how these models are trained and served. All the experts must exist in memory even though only a few run per token, which turns MoE into a distributed-systems problem as much as a machine-learning one. The trade MoE makes and the expert parallelism it forces.
MoE scales a model's parameters cheaply. But there's a second scaling axis that matters just as much for modern LLMs: context length — how much text the model can attend to at once. Attention's cost grows with the square of the sequence, and the memory to run it grows linearly and relentlessly, which is why long context was hard and why so much architectural ingenuity has gone into it.
MoE scales a model's parameters cheaply, but there's a second axis that matters just as much: context length. Attention's compute grows with the square of the sequence, and the KV-cache memory grows linearly and relentlessly — two distinct costs, often confused, that make long context hard. Understanding both is the setup for the efficiency techniques that solved them.
The two costs of long context — quadratic attention compute and linear KV-cache memory — each have a family of solutions, and together they're why modern models can handle context lengths that were impossible a few years ago. Grouped-query attention shrinks the KV cache; FlashAttention computes exact attention far faster; sliding-window and sparse patterns break the quadratic. This post covers the techniques that made long context practical.
The two costs of long context each have a family of solutions, and together they're why modern models handle context lengths that were impossible a few years ago. Grouped-query attention shrinks the KV cache; FlashAttention computes exact attention far faster; sliding-window and sparse patterns break the quadratic. The techniques that made long context practical, and which cost each attacks.
Put the pieces together and a modern frontier language model comes into focus: a deep transformer whose feed-forward blocks are Mixtures of Experts, whose attention is made efficient with grouped-query and FlashAttention, and which is engineered end to end around one goal — maximum capability per unit of compute and memory. This closing post assembles the anatomy and looks at where architecture is heading.
Put the pieces together and a modern frontier LLM comes into focus: a deep transformer whose feed-forward blocks are Mixtures of Experts, whose attention is made efficient with grouped-query and FlashAttention, engineered end to end around one goal — maximum capability per unit of compute and memory. This closing post assembles the anatomy and looks at where architecture is heading.
This series is part of a larger body of work by Pratik Dhanave, an Agentic AI Architect writing about production AI systems, distributed systems, and cloud-native engineering. Explore all course series, browse every post, or find topics via the tag index.