Distributed Caching
A cache inside a single application process is easy — but it doesn't scale, and every instance of your app has its own separate copy. The moment you run many application servers, you want a shared cache they all use, which means a cache that lives across the network on its own machines: a distributed cache like Redis or Memcached. This unlocks scale and sharing, but introduces the distributed-systems problems that a local cache never had. Understanding distributed caching is understanding how caching works at real scale.
Distributed caching spreads a cache across multiple machines — a shared, networked cache (like Redis or Memcached) that many application servers use. This post covers why distributed caches exist, how they work (a networked shared cache), how they scale across nodes (sharding and consistent hashing), and the distributed challenges. It builds on all the prior caching concepts, now at scale across machines, and connects caching to distributed systems.
Why distributed caching
Distributed caching — a cache running on separate machine(s), shared over the network by many clients — exists to solve the limits of a local (in-process) cache at scale:
- Local caches don’t scale or share. A local cache (inside one application process) is fast and simple, but has limits: it’s bounded by one machine’s memory (can’t grow beyond it), and each application instance has its own separate cache (not shared — many app servers mean many separate, inconsistent caches, each with low hit rate and duplicated data). At scale (many app servers, more data than one machine holds), local caches are insufficient. They don’t scale or share.
- A distributed cache is shared and scalable. A distributed cache runs on dedicated machine(s), accessed over the network by all the application servers — so it’s shared (all app instances use the same cache — consistent, higher combined hit rate, no duplication) and scalable (spread across multiple cache machines to hold more data than one machine could — the sharding below). It solves the local cache’s limits: one shared, scalable cache for the whole system. Shared + scalable is the point.
- Redis and Memcached: the standard tools. Redis and Memcached are the dominant distributed caching systems — in-memory data stores accessed over the network, used as shared caches by applications. Memcached is a simple, fast in-memory cache; Redis is a richer in-memory data store (more data structures, persistence options, features) also widely used as a cache. Both provide the shared, networked cache that applications use — the workhorses of distributed caching. (Redis is especially popular for its features beyond simple caching.)
Distributed caching — a shared, networked cache on dedicated machines (Redis, Memcached) — solves the local cache’s limits (not scalable beyond one machine, not shared across app instances) by providing one shared, scalable cache for the whole system. It’s how caching works at real scale. But moving the cache across the network changes things.
How a distributed cache works
A distributed cache is a networked, shared store — which introduces differences from a local cache worth understanding:
- Access is over the network. Unlike a local cache (in-process memory, nanoseconds to access), a distributed cache is accessed over the network — so access has network latency (much slower than local memory, though still much faster than the source database/computation it’s caching). A distributed cache is slower than local memory but faster than the source — it sits in the middle of the hierarchy. This network cost is a key difference: distributed cache hits aren’t free (network round-trip), though they’re still far cheaper than misses to the source. Fast, but not local-memory-fast.
- It’s a shared, separate service. The distributed cache is a separate service (its own machines) that all app servers talk to — shared state external to the application. This sharing is the benefit (one consistent cache) but makes the cache a dependency and potential bottleneck/single-point-of-failure (if the cache is down or slow, all clients are affected). The cache becomes infrastructure to run, scale, and keep available. It’s a service, with all that entails.
- It’s typically in-memory (fast but volatile). Distributed caches (Redis, Memcached) are in-memory (RAM) for speed — which makes them fast but volatile (data lost on restart/failure unless persisted — Redis offers persistence options, Memcached generally doesn’t). Cached data is generally disposable (it’s a cache — losable, refetchable from the source), so volatility is usually acceptable (a cache failure means misses, not data loss — the source has the data). In-memory speed with disposable data is the norm. Losing the cache is a performance hit, not a data-loss disaster (if used purely as a cache).
A distributed cache works as a networked, shared, in-memory service — slower than local memory (network latency) but far faster than the source, shared across app servers (a benefit but also a dependency/bottleneck), and typically volatile (in-memory, disposable cache data). These properties differ from a local cache and shape how it’s used. Its key capability is scaling across multiple machines.
Scaling across nodes: sharding and consistent hashing
To hold more data than one machine and scale, a distributed cache spreads data across multiple nodes — via sharding, made manageable by consistent hashing:
- Sharding: split data across nodes. Sharding (partitioning) splits the cached data across multiple cache machines (nodes) — each node holds a portion of the data. This lets the total cache hold more than any one machine (scaling capacity) and spread load across nodes (scaling throughput). To find an item, you determine which node holds it (typically by hashing the key to a node). Sharding is how a distributed cache scales beyond one machine — split the keyspace across nodes. Each key lives on one node.
- The rehashing problem. A naive sharding scheme (e.g.
node = hash(key) mod Nfor N nodes) has a serious problem: when you add or remove a node (N changes), almost all keys map to different nodes — so nearly the entire cache is invalidated/rehashed at once (a “cache stampede” of misses as everything must be refetched). Since nodes are added/removed regularly (scaling, failures), this massive rehashing on every change is unacceptable. Naive modulo sharding rehashes almost everything when the node count changes. This is the problem consistent hashing solves. - Consistent hashing: minimal rehashing. Consistent hashing is a technique that, when a node is added or removed, only requires a small fraction of keys to move (roughly 1/N) — not almost all of them. It maps keys and nodes onto a “ring” so that adding/removing a node only affects nearby keys, leaving most keys on their existing nodes. This makes scaling the cache (adding/removing nodes) cheap — minimal rehashing, minimal cache disruption. Consistent hashing is a foundational technique for distributed caches (and distributed systems generally), enabling elastic scaling without massive cache invalidation. It’s the key idea that makes sharded caches practical to scale.
Distributed caches scale by sharding (splitting data across nodes to hold more and spread load), made practical by consistent hashing (adding/removing a node moves only a small fraction of keys, not almost all — avoiding the massive rehashing of naive modulo sharding). Consistent hashing is the foundational technique enabling elastic distributed caches. Scaling across machines brings distributed-systems challenges.
Distributed caching challenges
Distributing the cache across machines introduces the distributed-systems challenges a local cache never had — worth knowing:
- Network latency and reliability. Distributed cache access crosses the network (latency) and the cache can be unreachable (network issues, cache down) — so clients must handle cache latency and failures (e.g. fall back to the source on cache errors — the cache-aside resilience). The network adds cost and failure modes absent in a local cache. Design for the cache being slow or unavailable. It’s a network dependency.
- Consistency across nodes and clients. With a shared cache used by many clients and sharded across nodes, consistency gets harder: the invalidation problem (from that post) now spans a distributed shared cache (invalidating across a shared cache used by many clients), and there are distributed-consistency subtleties. Keeping a distributed cache consistent is harder than a local one. Distribution compounds the invalidation challenge. (This connects to distributed-systems consistency broadly.)
- The cache as a critical dependency. A shared distributed cache becomes critical infrastructure — if it fails or is slow, all clients are affected (a potential single point of failure / bottleneck). So distributed caches need availability (replication, failover) and capacity management (scaling nodes), making the cache itself a system to operate reliably. The cache is no longer a local convenience but shared infrastructure with operational demands. Run it like the critical service it is.
- It’s a distributed system. Ultimately, a distributed cache is a distributed system, inheriting distributed-systems challenges (network, partial failure, consistency, scaling — from the distributed-systems series). Distributed caching sits at the intersection of caching and distributed systems, and understanding it means understanding both. The distributed part adds real complexity to the caching part. Caching at scale is distributed-systems engineering.
Distributed caching — a shared, networked cache (Redis, Memcached) scaled across nodes via sharding and consistent hashing — solves local caches’ scale and sharing limits but introduces distributed-systems challenges (network latency/failure, harder consistency, the cache as critical infrastructure). It’s how caching works at real scale, at the intersection of caching and distributed systems. Next: web and CDN caching — caching across the internet.
Key takeaways
- Distributed caching (a shared cache on dedicated machines, accessed over the network by many app servers — Redis, Memcached) solves local (in-process) caches’ limits: local caches can’t scale beyond one machine’s memory and aren’t shared (each app instance has a separate, inconsistent cache) — a distributed cache is shared (one consistent cache, higher combined hit rate) and scalable.
- A distributed cache is networked (slower than local memory due to network latency, but far faster than the source — it sits in the middle of the hierarchy), a shared separate service (a benefit but also a dependency/bottleneck/potential single point of failure), and typically in-memory (fast but volatile — though cache data is disposable/refetchable, so volatility is usually acceptable).
- It scales via sharding (splitting data across nodes to hold more and spread load — each key lives on one node, found by hashing) — but naive modulo sharding rehashes almost all keys when the node count changes (a stampede of misses), which is unacceptable given regular node changes.
- Consistent hashing solves this: adding/removing a node moves only a small fraction (~1/N) of keys, not almost all — enabling cheap, elastic scaling without massive cache invalidation, making it a foundational technique for distributed caches.
- Distributing the cache introduces distributed-systems challenges: network latency and reliability (handle cache slowness/failure, e.g. fall back to source), harder consistency (invalidation across a shared, sharded cache), and the cache as critical infrastructure (needs availability/failover and capacity management) — a distributed cache is a distributed system, at the intersection of caching and distributed systems.