Caching Pitfalls and Practice

This final post covers caching pitfalls — the failure modes that bite real systems (cache stampede, staleness bugs, penetration, and more) — and the practical wisdom of caching well, tying the series together. It’s the “what goes wrong and how to cache responsibly” post, synthesizing the series’ lessons into practical guidance. Because caching’s power comes with real, recurring pitfalls, knowing them is essential to caching well.

Cache stampede (thundering herd)

A classic, dangerous caching pitfall is the cache stampede (or “thundering herd”) — many requests hammering the source at once when a cache entry expires:

• What it is. When a popular cached item expires (or is evicted), the next requests all miss simultaneously — and all rush to the source to recompute/refetch it at once. A flood of simultaneous misses for the same hot item hits the source together — a “stampede” — potentially overwhelming the source (database, service) with a sudden spike of duplicate work. The stampede is many concurrent misses for the same expired hot key overwhelming the source. Counterintuitively, the cache expiring can cause an outage.
• Why it’s dangerous. The stampede can overload the source — the very thing the cache was protecting — causing a cascading failure (the source, hit by the stampede, slows or fails, which causes more misses, which worsens the stampede). A cache stampede on a hot item can take down the source (and thus the system), especially under high load. It’s a real, serious failure mode — the cache’s expiry becoming an outage trigger. Ironic and dangerous: the protection’s momentary absence causes the collapse.
• How to prevent it. Mitigations include: locking (only one request recomputes the expired item while others wait or serve stale — preventing the duplicate stampede), early/staggered expiration (refresh before expiry, or stagger expiry times so items don’t all expire together), and serving stale while revalidating (serve the old value while one request refreshes in the background). These prevent the simultaneous flood of misses from hitting the source. Preventing stampedes (especially for hot items) is important production caching. Don’t let a hot key’s expiry stampede the source.

Cache stampede (thundering herd) — many simultaneous misses for a popular expired item flooding the source at once, potentially overwhelming it and cascading to failure — is a classic dangerous pitfall, prevented by locking (one recompute), staggered/early expiration, and serve-stale-while-revalidating. It’s a case where the cache’s absence causes an outage. It’s not the only counterintuitive failure mode.

More pitfalls

Beyond stampede, several other caching pitfalls bite real systems — worth knowing to avoid them:

• Staleness bugs. The invalidation problem (from that post) in practice: cached data that’s stale (out of date) causing wrong behavior — users seeing old data, decisions on outdated values, inconsistencies. Staleness bugs are often subtle and intermittent (they appear only when the cache is stale, then vanish after refresh — hard to reproduce and debug). They’re a common, frustrating caching pitfall, stemming from the hard invalidation problem. Cache staleness is a top source of baffling bugs. When something’s “randomly wrong then fine,” suspect the cache.
• Cache penetration. When requests are for data that doesn’t exist (or isn’t cacheable), they always miss and always hit the source (nothing to cache) — so the cache provides no protection, and a flood of such requests (e.g. queries for non-existent keys, sometimes malicious) hits the source directly. Cache penetration is requests “penetrating” the cache to the source because the data isn’t cached. Mitigations include caching “not found” results (negative caching) or filtering invalid requests. Requests for non-existent data bypass the cache’s protection.
• Cold cache / cache warming. A cold (empty) cache — after a restart, deployment, or new cache — has no data, so everything misses until it fills, causing a burst of load on the source and slow responses initially. This cold-start problem can overload the source right after a restart. Cache warming (pre-populating the cache before serving traffic) mitigates it. A cold cache is temporarily as bad as no cache (plus miss overhead) — a real concern around restarts/deploys. Empty caches offer no protection until warmed.
• Over-caching and cache complexity. Caching too much, or caching things that shouldn’t be cached (rarely-reused or must-be-fresh data — from the fundamentals post), adds complexity and staleness risk for little benefit. And caching generally adds a layer of complexity (another system, invalidation logic, failure modes) that must be justified. Over-caching (or caching wrongly) causes more problems than it solves. Cache deliberately, not everywhere. More caching isn’t always better.

Caching pitfalls — staleness bugs (subtle, intermittent wrong data from stale caches — a top source of baffling bugs), cache penetration (requests for non-existent data bypassing the cache to the source), cold cache (empty cache after restart offering no protection until warmed), and over-caching (needless complexity/staleness) — bite real systems alongside stampede. Knowing them is essential to caching well. They all stem from caching’s inherent tradeoffs.

When not to cache

A crucial piece of caching wisdom, echoing the whole series: know when NOT to cache — because caching isn’t free, and inappropriate caching causes more harm than good:

• Don’t cache when the benefit is small. If data is rarely reused (low hit rate — little benefit) or cheap to obtain (caching saves little), caching adds complexity and failure modes for negligible gain. The performance benefit must justify caching’s cost (complexity, staleness risk). When the benefit is small, don’t cache. Caching low-value data is net-negative.
• Don’t cache when consistency is critical and data changes. If data must be perfectly current and changes frequently, caching causes staleness problems that outweigh the benefit (constant invalidation, or wrong data). For freshness-critical, frequently-changing data, caching is often the wrong tool. Don’t cache what you can’t keep consistent enough. Consistency-critical + changing = don’t cache.
• Don’t cache to avoid fixing the real problem. Sometimes caching is used to mask an underlying performance problem (a slow query, an inefficient system) that should be fixed instead. Caching a fundamentally-slow operation can hide (and entrench) a problem better solved at the source. Caching should complement good design, not paper over bad design. Fix the real problem when caching is just a band-aid over it. Don’t cache to avoid fixing what’s actually broken.
• The recurring theme: cache deliberately. As throughout the blog’s engineering series, use the right tool for the need — cache when the benefit (reused, expensive, tolerant data) justifies the cost, and don’t when it doesn’t. Reaching for caching reflexively (because it’s a common technique) without weighing the tradeoff is a mistake. Cache deliberately, where it genuinely helps. Caching is a powerful tool for specific situations, not a default. Deliberate, justified caching.

Knowing when not to cache — small benefit (rarely-reused/cheap data), consistency-critical changing data, or masking a fixable problem — is essential caching wisdom: cache deliberately where the benefit justifies the cost, not reflexively. This “right tool for the need” discipline recurs across the blog’s engineering series. It’s the counterweight to caching’s appeal.

Caching well: the practical wisdom

To close the series, the distilled practical wisdom of caching well:

• Maximize the hit rate on the right data. Cache the hot, expensive, staleness-tolerant data (the fundamentals post) to get a high hit rate (the key metric) — the core of effective caching. Cache the right things, in an adequately-sized cache, with a good eviction policy (LRU by default). Effective caching is high-hit-rate caching on well-chosen data. Get the hit rate right.
• Handle invalidation by choosing a consistency point. Accept that cache invalidation is hard (the famous problem) and choose a point on the consistency spectrum — usually TTL/bounded staleness where data tolerates it (simple, robust), explicit invalidation only where freshness truly requires it. Design for bounded staleness, not perfect consistency. Navigate the consistency tradeoff deliberately; don’t chase impossible perfection. Bounded staleness is usually enough.
• Anticipate the failure modes. Design for the pitfalls: prevent stampedes (locking, staggered expiry, serve-stale) on hot items, handle cold caches (warming) around restarts, guard against penetration (negative caching), and expect staleness bugs (design to tolerate staleness). Anticipating these failure modes is what separates robust production caching from caching that breaks under load or causes baffling bugs. Expect and design for what goes wrong.
• Cache deliberately, and remember it’s hard. Ultimately: cache deliberately (where the benefit justifies the cost, not reflexively), understand it’s a tradeoff (performance for complexity and consistency risk), and respect that the two genuinely hard things (cache invalidation, naming) are hard. Caching is one of the most powerful performance techniques in computing — and one of the most subtle to get right. Used well (right data, chosen consistency, anticipated failures, deliberate application), it makes systems dramatically faster; used carelessly, it causes subtle, painful problems. Cache with respect for its power and its difficulty.

Caching well means maximizing the hit rate on the right (hot, expensive, tolerant) data, navigating the hard invalidation problem by choosing a consistency point (usually bounded staleness/TTL), anticipating the failure modes (stampede, cold cache, penetration, staleness), and caching deliberately where the benefit justifies the cost. That completes the series: from why caching exists (locality, the memory hierarchy) through fundamentals, eviction, invalidation, patterns, distributed caching, and web/CDN caching, to pitfalls and practice. Caching is a universal, powerful, and genuinely subtle technique — respect both its power and its difficulty, and cache deliberately.

Key takeaways

• Cache stampede (thundering herd) — many simultaneous misses for a popular expired item all flooding the source at once, potentially overwhelming it and cascading to failure (the cache’s momentary absence causing an outage) — is prevented by locking (one request recomputes), staggered/early expiration, and serving stale while revalidating.
• Other pitfalls bite real systems: staleness bugs (subtle, intermittent wrong data from stale caches — a top source of baffling “randomly wrong then fine” bugs), cache penetration (requests for non-existent data always missing and hitting the source — mitigate with negative caching), cold cache (empty after restart, no protection until warmed — mitigate with cache warming), and over-caching (needless complexity/staleness for little benefit).
• Know when NOT to cache: when the benefit is small (rarely-reused or cheap data), when consistency is critical and data changes frequently, or when caching would mask a fixable underlying problem rather than complement good design — cache deliberately, not reflexively (the recurring “right tool for the need” discipline).
• Cache well by maximizing the hit rate on the right data (hot, expensive, staleness-tolerant — adequately sized, LRU eviction), and by handling invalidation through choosing a consistency point (usually bounded staleness/TTL where tolerable, explicit invalidation only where freshness truly requires it — design for bounded staleness, not impossible perfect consistency).
• Anticipate the failure modes (prevent stampedes, warm cold caches, guard penetration, tolerate staleness) — the difference between robust production caching and caching that breaks under load or causes baffling bugs — and cache deliberately, respecting that caching is one of computing’s most powerful performance techniques and one of the most subtle to get right (the two genuinely hard things stay hard).

Further reading

• Cache stampede (Wikipedia)
• Cache (computing) — caching in practice (Wikipedia)
• Web and CDN caching (previous post)