Start here: a cache is a second copy of the truth#
TL;DRthe 30-second version
- A cache stores a copy of an answer so future reads skip the slow, expensive source. It trades speed for two costs: the data can be stale, and you now maintain a second copy of the truth.
- The whole payoff lives in the hit ratio — the fraction of reads the cache answers itself. A high hit ratio means most reads are fast; a low hit ratio means you pay the cache's cost and get almost none of its benefit.
- Caching pays when reads dominate writes, the same data is read over and over (locality), the source is slow or expensive, and some staleness is acceptable.
- Caching hurts when data changes on nearly every read, reads must be perfectly fresh, the working set is far bigger than the cache (thrash), or you're using it to hide a source you should just fix.
Reach for a cache and the mental picture is simple: something is slow, so keep the answer nearby and hand it back fast next time. That picture is right, and it's exactly why caches get sprinkled everywhere. The part that's missing is the bill. A cache is a second store of data that has to agree with the first one, and data has an inconvenient habit of changing. The instant the source changes and the cache doesn't, your two copies disagree, and a reader gets an answer that used to be true.
So the decision is never 'is a cache faster' — a cache is almost always faster on a hit. The decision is whether your workload turns that speed into a real win without the staleness and upkeep eating the gain. That depends on facts about your data: how often it's read versus written, whether the same items are read repeatedly, how slow the real source is, and how fresh the answer has to be. Get those facts wrong and a cache adds latency, bugs, and infrastructure while barely lifting a finger to help.
Hit ratio is the whole game#
A cache sits in front of a slower source. A read checks the cache first. If the item is there, that's a hit and the reader gets it fast. If it isn't, that's a miss: the read falls through to the source, pays the slow path, and usually stores the result in the cache on the way back so the next reader hits. The fraction of reads that are hits is the hit ratio, and it is the single number that decides whether a cache is worth anything.
Here's why. The latency a reader actually feels is a weighted average of the fast path and the slow path. Say a cache hit takes 1 ms and a miss takes 50 ms (the cache lookup plus the slow source). At a 95% hit ratio, the average read is 0.95 × 1 ms + 0.05 × 50 ms = 3.45 ms — nearly the speed of the cache. Now drop the hit ratio to 50%: 0.5 × 1 ms + 0.5 × 50 ms = 25.5 ms. Same cache, same source, but the average read is more than seven times slower, because half the reads pay full price and the miss is expensive. The cache didn't get worse; the workload did.
That formula is the entire case for caching, and it tells you exactly what to look for. You want a high hit ratio, and a high hit ratio needs two things from your data. First, reads have to outnumber writes, because a write usually changes or invalidates the cached copy — a workload that writes as often as it reads keeps knocking the cache out from under you. Second, reads have to concentrate on a small, hot subset that's read over and over, so the cache can hold that subset and answer most requests from it. That concentration has a name.
The cost of a cache#
A cache on a hit is nearly free. Everything expensive about a cache happens around the misses and the writes. These costs don't show up in a demo; they show up in production, and they're the reason 'just add a cache' is a bigger decision than it looks.
- Cache invalidation: when the source changes, the cached copy is now wrong, and something has to notice and fix it — by deleting the entry, overwriting it, or letting it expire on a timer (TTL, time-to-live). Getting this right for every way data can change is genuinely hard, and getting it wrong is how stale data reaches users.
- Staleness bugs: a reader sees an old value — a price that already changed, a permission that was just revoked, a post that was deleted. Whether that's harmless or a serious bug depends entirely on the data, and it is the cost you trade speed for.
- The thundering herd (cache stampede): when a popular entry expires or is evicted, every request that wanted it misses at once and stampedes the source together. The one slow query you were caching now fires a thousand times in a burst — and can knock over the very source the cache was protecting.
- Extra infrastructure and failure modes: a cache is another system to run, size, monitor, and pay for. It can go down, run out of memory, or partition away — and now you have to decide whether a cache outage means slow (fall through to the source) or down (fail the request).
- Hiding a fixable problem: the most insidious cost. A cache can paper over a source that's slow because of a missing index, an N+1 query, or a bad schema. You've spent a cache to avoid fixing a one-line problem, and the slowness is still there waiting for the first cache miss.
PredictYou put a 60-second cache in front of a dashboard query to cut load. It works for a week, then one afternoon the database gets hammered by a flood of identical queries and slows to a crawl. What most likely happened, and what class of problem is it?
Hint: What happens to a popular entry the moment its timer runs out, if many readers want it at once?
A cache stampede (thundering herd). The cached dashboard entry expired, and because the dashboard is popular, every request that wanted it missed at the same instant and fell through to the database together — so instead of one query refreshing the cache, a thousand identical queries hit the source in a burst. It is the classic failure mode of a naive time-based cache under load: the cache protected the database right up until the moment the entry expired, and then it protected nothing. The fixes are stampede controls — have only one request recompute the value while the others wait or serve the old copy (request coalescing / locking), or refresh the entry slightly before it expires so it never fully lapses under traffic. The one-line takeaway: a cache miss on a hot key isn't one slow read, it's a synchronized crowd.
When to cache, and when not to#
Put the payoff and the costs together and the decision comes down to four questions about your data. None of them is about the cache; all of them are about the workload. A cache pays when the answers line up, and it hurts when they don't. First, the cases where it pays:
- Reads dominate writes: the more reads per write, the longer a cached copy stays useful before a write invalidates it. A read-heavy workload is the classic fit.
- There's locality: a hot subset is read over and over, so a small cache catches most reads and the hit ratio stays high.
- The source is slow or expensive: a heavy aggregation query, a call to a remote service, a rate-limited third-party API — the bigger the gap between a hit and a miss, the more each hit saves.
- Some staleness is acceptable: the data can be a few seconds or minutes behind without harm. A view count, a recommendation, a rendered page — being slightly old is fine.
Flip each of those and you get the cases where a cache is all cost and no benefit. This is the half candidates skip, and naming it unprompted is what separates a strong answer:
- Data changes on almost every read: if each read wants a value a write just changed, the hit ratio is near zero — every read misses, pays the cache lookup, then pays the source. You've added a layer that only ever slows things down.
- Reads must be perfectly fresh: a balance during a transfer, inventory at checkout, a permission check — anywhere a stale answer is a correctness bug, not a cosmetic one. Don't cache what you can't afford to be wrong.
- The working set is far bigger than the cache: if reads spread across far more data than the cache can hold, entries get evicted before they're reused. The cache churns (thrashes), the hit ratio stays low, and you pay to constantly refill it.
- Writes dominate: a write-heavy workload spends its time invalidating cached entries faster than reads can benefit from them, so the cache is mostly overhead.
PredictA service looks up a user's current account balance on every request, and balances change constantly as transactions post. An engineer proposes caching the balance to reduce database load. What's wrong with this, on two separate grounds?
Hint: Think about the hit ratio when data changes every read, and separately about what a stale balance means.
It fails on both the payoff and the correctness axis. On payoff: balances change constantly, so the hit ratio is near zero — nearly every read wants a value a recent write just changed, meaning almost every read misses, pays the cache lookup, and then pays the database anyway. You've added latency and a second system for essentially no hit-rate benefit. On correctness: a balance is exactly the kind of data that must be perfectly fresh — showing a stale balance (or worse, deciding whether a transaction is allowed against one) is a real bug, not a cosmetic delay. So this is a double 'don't cache': the workload has no locality to exploit and the data can't tolerate staleness. The right move is to make the source fast (index the query, precompute where safe), not to cache a value you can neither keep warm nor allow to be stale.
Where to cache: the layers#
'Should I cache' is really two questions, and the second is 'where'. A cache can live at several points between the reader and the source, and each layer trades freshness, hit ratio, and blast radius differently. The closer the cache sits to the reader, the faster the hit and the more readers it serves — but the harder it is to invalidate, because you no longer control the copy. The closer it sits to the source, the easier it is to keep fresh and share, but the less latency it saves.
- Client / browser cache: the copy lives on the user's device. A hit costs zero network — the fastest possible — and takes all the load off your servers. The catch: you can't reach out and invalidate it, so you control staleness only by setting how long it's allowed to live. Best for data that's private to one user and tolerant of being a little old.
- CDN / edge cache: copies held at points of presence near users around the world. A CDN absorbs enormous read volume for shared, cacheable content (images, scripts, whole pages) close to the reader. It has its own topic — see the CDN page — and it shines for the same content served to many people.
- Application in-process (near cache): a cache inside the application server's own memory. Hits are extremely fast (no network hop at all), but each server has its own copy, so servers can disagree with each other, and the cache is lost when the process restarts. Good for small, hot, read-mostly reference data.
- Shared distributed cache (Redis, Memcached): a separate cache service every application server talks to over the network. One copy shared across all of them, so it's consistent between servers and survives an app restart — at the cost of a network hop per lookup and another system to run. This is the default when people say 'add a cache'.
These aren't exclusive — a real system often stacks them (browser, then CDN, then a shared Redis in front of the database). The point for a decision is that 'where' changes the trade: a browser cache buys the most speed and the least control over freshness; a shared cache buys the least speed and the most control. Pick the layer whose freshness and blast-radius trade matches the data.
The decision, in two tables#
The whole decision fits in a table. Read down the left column for a property of your workload, and it tells you whether a cache is a fit.
| If your workload… | Cache? | Because |
|---|---|---|
| Reads dominate writes, with a hot subset | Yes | High hit ratio — most reads served from the fast copy |
| Source is slow or expensive per read | Yes | Each hit skips a big cost; the miss penalty is what you're avoiding |
| Some staleness is acceptable | Yes | You can serve a slightly old copy without a correctness bug |
| Data changes on nearly every read | No | Hit ratio near zero — all cost, no benefit |
| Reads must be perfectly fresh | No | A stale answer here is a correctness bug, not a delay |
| Working set far bigger than the cache | No | Entries evicted before reuse — the cache thrashes |
| Source is slow for a fixable reason | Fix first | A cache hides a missing index; the slowness waits at the next miss |
If the first table says yes, the second decides where — because each layer trades speed against control over freshness.
| Cache layer | Speed on hit | Freshness control | Best for |
|---|---|---|---|
| Browser / client | Fastest (no network) | Weakest — TTL only | Per-user, staleness-tolerant data |
| CDN / edge | Very fast (near user) | Moderate | Shared, cacheable content at scale |
| In-process (near cache) | Very fast (no hop) | Per-server, can diverge | Small hot reference data |
| Shared cache (Redis) | Fast (one network hop) | Strongest — one shared copy | The general-purpose default |
Two tables, one message: the first decides whether to cache at all, the second decides where. If the first says no, no clever choice of layer rescues it — a cache in the wrong place for the wrong workload is just latency and a bill.
In the wild
- CDNs (Cloudflare, Akamai, Fastly) are caches at the edge: they hold copies of cacheable content near users, absorbing the bulk of read traffic for high-traffic sites so the origin never sees it. A classic high-read, high-locality, staleness-tolerant fit.
- Redis and Memcached are the shared distributed cache almost every large web app runs in front of its database — caching session data, rendered fragments, hot rows, and expensive query results that are read far more than they change.
- A CPU's L1/L2/L3 caches are the same idea in hardware: small, fast copies of main memory that work only because programs have strong locality — the same addresses read over and over. When a program has poor locality, those caches thrash and performance craters, the hardware version of a low hit ratio.
- Database buffer pools and page caches (Postgres shared buffers, the OS page cache) hold hot disk pages in RAM. They're why a repeated query is fast the second time — see the Page Cache topic — and why the first cold read after a restart is slow.
- Content sites cache rendered pages and API responses with a short TTL: a news homepage read millions of times that changes every few minutes is the textbook case — a huge read/write ratio, strong locality, and seconds of staleness invisible to readers.
Pitfalls & gotchas
Isn't a cache basically always a good idea for read-heavy systems?
Only when read-heavy also means high locality and tolerable staleness. Read-heavy with no hot subset (every read hits a different item) gives a low hit ratio, so the cache barely helps. And read-heavy data that must be perfectly fresh — a balance, a permission — shouldn't be cached at all. Read-heavy is necessary but not sufficient; you also need locality and staleness tolerance.
What's the difference between a low hit ratio and a cache stampede?
A low hit ratio is a steady-state problem: your workload just doesn't have the locality or read/write ratio to fill the cache with useful entries, so most reads miss all the time. A stampede is a burst problem: the hit ratio is fine normally, but when a hot entry expires, all the readers that wanted it miss at the same instant and flood the source together. Different causes, different fixes — the first says 'don't cache this', the second says 'add stampede control'.
How do I pick a TTL (time-to-live)?
The TTL is you setting a price on staleness. A short TTL keeps data fresh but lowers the hit ratio (entries expire before they're reused) and invites stampedes; a long TTL raises the hit ratio but serves staler data. Pick it from how stale the data is allowed to be, not from how much you want to cache. If no TTL is short enough to be safe, that's a sign the data shouldn't be cached on a timer at all — invalidate it explicitly on write instead.
Why not just cache everything to be safe?
Because a cache isn't free even when it isn't helping. Every cached item costs memory, and every write has to invalidate its cached copies — so caching data that's rarely reread or frequently written adds invalidation work and evicts the genuinely hot entries you needed room for. Caching everything lowers your hit ratio on the things that mattered. Cache the hot, read-mostly, staleness-tolerant subset; leave the rest to the source.
QuizA team caches every row their API reads, with a long TTL, to 'reduce database load everywhere.' Load drops at first, then complaints about stale data climb and the cache's memory keeps filling. What did they most likely get wrong?
- The TTL was too short, so nothing stayed cached.
- They cached indiscriminately — including write-heavy and freshness-critical data — so they got staleness bugs and a low effective hit ratio while evicting the genuinely hot entries.
- Redis is the wrong technology for this; they should use a CDN.
- Caching can never reduce database load, so the initial drop was a measurement error.
Show answer
They cached indiscriminately — including write-heavy and freshness-critical data — so they got staleness bugs and a low effective hit ratio while evicting the genuinely hot entries. — Caching everything is its own failure. A long TTL on data that changes often produces exactly the stale-data complaints they're seeing, and filling the cache with cold or write-heavy rows wastes memory and evicts the hot, read-mostly entries that actually deserved the space — dragging the effective hit ratio down. The fix isn't a different TTL or a different product; it's selectivity: cache the subset that's read far more than it's written and can tolerate being slightly old, and invalidate on write (or don't cache) the data that must stay fresh. A cache is a scalpel for the hot path, not a blanket.
In an interview
This comes up constantly, and the trap is reaching for a cache as a reflex. Interviewers want to see you treat it as a trade with a cost, name the conditions that make it pay, and — the real signal — name the cases where you would not cache. Don't just say 'add a cache'; say what has to be true for it to help.
- Frame it as a trade first: a cache buys speed by keeping a second copy, and pays for it in staleness and the work of keeping the copy honest (invalidation). Lead with that, not with 'add Redis'.
- Name the four conditions that make it pay: reads dominate writes, there's locality (a hot subset), the source is slow or expensive, and some staleness is acceptable. Tie each back to the hit ratio.
- Name when NOT to cache, unprompted: data that changes every read (hit ratio near zero), reads that must be perfectly fresh (correctness), a working set that doesn't fit (thrash), and write-heavy workloads. This is the part that shows judgment.
- Watch for the anti-pattern of caching to hide a fixable source (a missing index, an N+1 query). Say you'd fix the source first, then cache what's still hot.
- Know the failure modes by name: the cache stampede / thundering herd on a miss, and stale reads on invalidation — plus the mitigations (request coalescing, refresh-ahead, explicit invalidation on write, a TTL chosen from the staleness budget).
- Say where you'd cache and why: browser or CDN for shared, staleness-tolerant content near the user; a shared Redis or Memcached in front of the database for hot rows and expensive queries. Connect it to cache strategies (cache-aside, write-through) and the eviction policy (LRU/LFU).
PredictAn interviewer says: 'Our product page is slow — it runs a heavy query joining inventory, pricing, and reviews on every page load, and traffic is spiking.' They ask if you'd add a cache. What's the strong answer?
Hint: Does every field on that page have the same tolerance for being stale?
Don't answer yes or no — separate the pieces by how fresh each has to be, because they don't share a staleness budget. Reviews and the rendered page body are read constantly, change slowly, and tolerate being a few minutes old: perfect for a cache (or a CDN at the edge), with a short TTL and a huge read/write ratio giving a high hit ratio. Inventory and price are the opposite — a stale 'in stock' or an old price is a real correctness problem at checkout, so those you keep fresh, either from the source or with explicit invalidation on write, not a lazy TTL. So the strong answer is: cache the heavy, staleness-tolerant parts aggressively (that's where the read load and locality are), keep the freshness-critical fields live, and — before any of it — check whether the join is slow for a fixable reason (a missing index) rather than caching around a problem you should fix. Naming that one page mixes cacheable and un-cacheable data, instead of blanket-caching the whole thing, is the signal.
References
- Designing Data-Intensive Applications, Ch. 1 & 3 — Caches as derived, denormalized data you have to keep in sync with the source of truth.
- Scaling Memcache at Facebook (NSDI 2013) — A production look at hit ratio, thundering-herd stampedes, and invalidation at scale.
- Redis — Key eviction & policies — How a shared cache manages a bounded working set — LRU/LFU eviction when memory fills.
- MDN — HTTP caching — How browser and CDN caches use TTLs and validation to trade freshness for speed.
- Cache stampede — The thundering-herd failure on expiry, and the standard mitigations (locking, refresh-ahead).
Ready to try it?
The simulator is a real, deterministic implementation — pick a scenario and step through it, scrubbing the timeline forward and backward through every change.