Caching 101: Why Facebook Keeps So Much in Memcached (and How to Cache Without Breaking Things)
By Devang Pandya · Cloud Tech by Devang · Cloud • DevOps • AI — Made Simple
This post is the full breakdown for the Bapu ki Class Reel on caching. If you came here after commenting CACHE, welcome. Bapu explained it with his chashma on the table; here is the full engineering version.
Your database is at 100% CPU. Pages are slow. You look at the queries and notice something: most of them are reading the same data again and again. The same user profiles, the same product details, the same home page rows.
This is the moment for a cache.
Caching is one of the most powerful tools in system design, and one of the easiest to get wrong. In interviews it often comes as a scenario like the one Beta brings to Bapu in the Reel:
"Your database hits 100% CPU on repeated reads. Fix it."
In this post I will explain, in simple English, what a cache is, how Facebook used memcache at enormous scale, the cache-aside pattern, why you should delete cache keys on write instead of updating them, how TTLs work, and how to stop a cache stampede. These are the exact points that separate a junior answer from a senior one. In 14+ years of running production systems, I have seen caches save databases, and I have seen badly designed caches take them down. Let's do it right.
1. What a cache is
A cache is a fast, temporary copy of data, kept close to where it is needed. Usually that means in memory (RAM), which is far faster than reading from a database that has to parse a query, find rows and maybe read disk.
Popular in-memory caches are Memcached and Redis. On AWS, Amazon ElastiCache runs both as a managed service.
The idea is simple:
- Most reads hit the cache and return in well under a millisecond inside the data centre.
- Only cache misses go to the database.
- The database now handles a small share of the reads, so its CPU drops.
The share of reads served from the cache is called the hit ratio. A high hit ratio is what makes caching worth it.
2. Facebook's memcache: caching at extreme scale
In 2013, Facebook engineers published a well-known paper, "Scaling Memcache at Facebook", at the USENIX NSDI conference. It describes how they used memcached as a huge caching layer in front of their databases.
Some facts from the paper:
- The system handled billions of requests per second and stored trillions of items.
- It served over a billion users around the world.
- Memcache was used as a demand-filled look-aside cache: web servers check the cache first and fill it from the database on a miss.
- On writes, web servers updated the database and then sent a delete to memcache.
- They used leases to solve stale data and thundering herd problems.
- They had a Gutter pool of servers to take over when a few memcached servers failed, so the databases would not get flooded.
- They used an invalidation daemon called mcsqueal that reads committed database changes and sends deletes to the caches.
You do not need Facebook's scale to use these ideas. They work the same for a startup with one Redis node.
3. The cache-aside pattern (look-aside)
This is the most common caching pattern, and it is what Facebook's paper calls a demand-filled look-aside cache.
On a read
- The app asks the cache for the key, for example
user:42. - Hit: the cache has it. Return it. Done.
- Miss: the cache does not have it. The app reads from the database.
- The app puts the result in the cache, so the next read is a hit.
- Return the data.
def get_user(user_id):
key = f"user:{user_id}"
user = cache.get(key)
if user is None: # cache miss
user = db.query_user(user_id)
cache.set(key, user, ttl=300) # keep for 5 minutes
return user
Why it is popular
- Simple. The app is in control. The cache is "just" a fast copy.
- Lazy. Only data that is actually read gets cached.
- Safe on failure. If the cache goes down, the app can still read from the database (slower, but correct). You must make sure the database can survive that extra load, though.
Microsoft's Azure Architecture Center documents this as the Cache-Aside pattern, and it applies equally to Redis, Memcached, ElastiCache or any other cache.
4. On a write: delete, don't update
When data changes, the cache copy becomes stale. What should you do?
The Facebook pattern is:
- Update the database (the source of truth).
- Delete the cache key.
- The next read misses, reads the fresh value from the database and fills the cache again.
Why delete instead of writing the new value into the cache? The Facebook paper explains that they chose to delete because deletes are idempotent: doing them twice is harmless. Updating the cache directly opens the door to race conditions. Two writers can update the database in one order and the cache in the other order, and you end up with an old value stuck in the cache.
Remember too that the cache is not the source of truth. It is allowed to lose data. The database is the authority.
5. TTL: let old data expire
A TTL (time to live) tells the cache to remove a key after a set time, for example 5 minutes.
TTLs protect you from mistakes:
- If an invalidation is missed, the stale value still disappears after the TTL.
- Data that nobody reads anymore does not stay in memory forever.
How long should a TTL be? It depends on how stale the data is allowed to be:
- A product description: maybe minutes or hours.
- A stock price or account balance: very short, or do not cache it at all.
- A user's own profile right after they edit it: delete on write so they see their change immediately.
A useful trick is to add a little random jitter to TTLs, so thousands of keys do not all expire at the same second.
6. The cache stampede (thundering herd)
Here is the trap that takes down databases.
Imagine a very popular key, like the home page data. Its TTL expires, or it gets deleted because of an update. In that instant, thousands of requests miss the cache at the same time. All of them go to the database for the same data. The database, which was relaxed a second ago, suddenly gets a flood. This is called a cache stampede or thundering herd.
How Facebook stopped it: leases
The paper describes leases. When a client misses the cache, memcached gives it a lease token, a permission to fill that key. Memcached hands out a token for a given key only once every 10 seconds by default. Other clients that miss during that time are told to wait a short moment and retry. Usually the lease holder fills the cache within a few milliseconds, so the retries become hits.
The paper reports a striking result: for a set of keys prone to thundering herds, the peak database query rate dropped from 17,000 per second without leases to 1,300 per second with leases.
How you can stop it
- Locking / single flight: only one request rebuilds a missing key. Others wait or get the old value.
- Serve stale while revalidating: keep serving the slightly old value while one request refreshes it in the background.
- Early refresh: refresh hot keys a little before their TTL ends.
- TTL jitter: avoid many keys expiring together.
7. Other things to plan for
- What to cache: data that is read often and changes rarely. Do not cache everything.
- Cache sizing and eviction: when memory is full, the cache removes keys, usually the least recently used (LRU).
- Cold start: a new or restarted cache is empty, and every request misses. Warm it up gradually.
- Cache failure: if a cache node dies, its keys go to the database. Facebook's Gutter pool is one answer; replicas and gradual failover are others.
- Hot keys: a single extremely popular key can overload one cache node. Replicate it or use a small local in-process cache in front.
8. The example visuals
The "database at 100% CPU on repeated reads" line is an example scenario, the way interviewers phrase it. The App, Cache and Database boxes in the Reel are simplified. The Facebook facts (billions of requests per second, trillions of items, look-aside caching, delete on write because deletes are idempotent, leases for stale sets and thundering herds, the 10-second token rate, 17K/s to 1.3K/s, Gutter and mcsqueal) are verified from the NSDI 2013 paper. Facebook's current infrastructure may differ from what the 2013 paper describes.
9. How to answer this in an interview
Question: "Your database hits 100% CPU on repeated reads. Fix it."
Strong answer:
"Since most queries read the same data, I would add an in-memory cache like Redis or Memcached in front of the database, using the cache-aside pattern. On a read, check the cache first. On a miss, read the database and fill the cache. On a write, update the database and then delete the cache key, rather than updating it, because deletes are idempotent and avoid race conditions. That is exactly what Facebook described in its memcache paper, where the system handled billions of requests per second. I would set TTLs based on how stale each type of data can be, with some jitter. And I would protect against cache stampedes, when a hot key expires and many requests hit the database at once, using locks or leases, like Facebook did. I would measure the hit ratio and make sure the database can survive if the cache fails."
Follow-ups to expect:
- Cache-aside vs write-through? Cache-aside fills on read; write-through writes to the cache and database together on every write.
- Why delete instead of update on write? Deletes are idempotent and avoid stale values from reordered updates.
- What is a cache stampede? Many requests miss a hot key at once and flood the database. Use locks, leases or stale-while-revalidate.
- Redis or Memcached? Memcached is a simple, fast key-value cache. Redis adds data structures, persistence options and replication.
Sources
- Nishtala et al., "Scaling Memcache at Facebook", USENIX NSDI 2013 (billions of requests per second, trillions of items; demand-filled look-aside cache; delete on write because deletes are idempotent; leases for stale sets and thundering herds, one token per key every 10 seconds, 17K/s to 1.3K/s; Gutter; mcsqueal): https://www.usenix.org/system/files/conference/nsdi13/nsdi13-final170_update.pdf
- Microsoft Azure Architecture Center, "Cache-Aside pattern": https://learn.microsoft.com/en-us/azure/architecture/patterns/cache-aside
- AWS, "Amazon ElastiCache" (managed Redis/Valkey and Memcached): https://aws.amazon.com/elasticache/
Preparing for a Cloud, DevOps or SRE interview? Book a Free 15-min Cloud & DevOps Interview Guidance Call with me: what to study, how to answer system-design and on-call questions, plus a CV and roadmap check 👉 topmate.io/cloudtechbydevang
About the author
- I help engineers learn cloud, DevOps, SRE and AI in simple English, crack interviews and follow clear roadmaps. 14+ years building and running platforms on AWS, GCP, Kubernetes and Terraform. One topic a day.
Comments
Join the discussionNo comments yet. Be the first to comment.
