Your Cache Is Load-Bearing and Nobody Sized the Database for a Miss

A product feed endpoint I helped debug sat at 40ms p99 for about a year. One Tuesday afternoon it went to 12 seconds, and it stayed there for twenty minutes after traffic had already dropped back to normal levels. Nobody had deployed anything. No instance died. A Redis key reached its TTL, exactly as designed, and the database fell over.

The uncomfortable version of this story is that the system had been broken the whole time. We just never ran the code path that proved it.

The miss path nobody measured

Every cached endpoint has two latencies, and most dashboards show you one of them. The hit path was 40ms. The miss path was an 800ms query with four joins against the primary, and nobody had ever asked what happens when a few thousand requests take that path in the same second.

At peak the service was doing roughly 40k requests per second across all pods, and the feed key was shared by every logged out visitor. One key. Five minute TTL. When it expired, every in-flight request missed at once and every one of them decided to go rebuild it.

Nine thousand identical queries, all computing the same rows, all writing the same value back. The database was not overloaded in any interesting sense. It was doing the same work nine thousand times because nothing in the system said "someone is already handling this."

Why it stayed down after the traffic left

This is the part that surprises people the first time they see it. Traffic dropped back to a level the system had handled comfortably for a year, and the system stayed broken.

The mechanics are dull and mechanical. Each queued request holds a connection from the pool while it waits. The client times out at 10 seconds and gives up, but Postgres keeps running the query to completion, so the pool stays full doing work nobody will read. The client retries, which adds a fresh request on top. Now you have a feedback loop where the load generated by the failure exceeds the load that caused it.

That is a metastable failure: the same input load has two stable states, a good one and a bad one, and you got knocked into the bad one. It does not recover on its own because the badness is self-sustaining. We got out with a rolling restart, which is a polite way of saying we dumped every queued request on the floor.

Treat expiry as a hint, not a cliff

The fix is not a longer TTL. A longer TTL means the same stampede, less often, with staler data in between.

The move is to split expiry into two timestamps. At the soft TTL the value is old enough to want refreshing. At the hard TTL it is genuinely unusable. Between those two points, readers get the stale value immediately while exactly one of them goes and rebuilds it.

In Node that means two layers of collapsing. A per-process map kills duplicate work inside a pod, and a Redis lock kills it across pods.

const inflight = new Map<string, Promise<Feed>>();

async function getFeed(key: string): Promise<Feed> {
  const raw = await redis.get(key);
  if (raw) {
    const entry = JSON.parse(raw) as Entry<Feed>;
    if (Date.now() < entry.softExpiresAt) return entry.value;
    void revalidate(key).catch(() => {});  // never awaited
    return entry.value;                    // stale, but now
  }
  return revalidate(key);                  // cold, has to block
}

function revalidate(key: string): Promise<Feed> {
  const existing = inflight.get(key);
  if (existing) return existing;

  const p = (async () => {
    const got = await redis.set(`lock:${key}`, podId, { NX: true, PX: 10_000 });
    if (!got) {
      await sleep(50 + Math.random() * 150);
      const raw = await redis.get(key);
      if (raw) return (JSON.parse(raw) as Entry<Feed>).value;
    }
    const value = await db.buildFeed();
    const ttl = 300_000;
    await redis.set(key, JSON.stringify({
      value,
      softExpiresAt: Date.now() + ttl * (0.8 + Math.random() * 0.2),
    }), { PX: ttl * 3 });
    return value;
  })().finally(() => inflight.delete(key));

  inflight.set(key, p);
  return p;
}

The jitter on softExpiresAt matters more than it looks. Without it, keys written during the same incident expire during the same second forever after, and you have accidentally built a synchronized herd that reconvenes every five minutes.

What you give up

Stale reads, obviously. You should be able to say how stale in seconds, because "eventually consistent" is not a number anyone can design against. Ours was bounded by the refresh interval plus the rebuild time, so about six seconds worst case, which the product owner was fine with once someone actually asked.

The lock is a new thing that can break. If a pod dies holding it, the key stays stale until the lock TTL expires. Set that lock TTL a bit above your p99 rebuild time and no higher, or a crash during deploy turns into ten minutes of frozen data.

Cold misses still stampede, and soft TTLs do nothing for them. A Redis failover, a new key going live, a cache flush, all of those hand you an empty cache with full traffic pointed at it. That case needs a hard concurrency limit on the origin call and a 503 with Retry-After for everyone past it. Queueing those requests just moves the pileup one layer up.

Measure the path that hurts

Hit ratio on its own tells you almost nothing. A 99% hit ratio with one shared key is more dangerous than 85% spread over thousands of keys, because the 1% arrives all at once.

Track origin queries per second per cache key and alert on it directly. That number is the one that killed us, and it was not on any graph at the time. Record rebuild duration as its own metric rather than folding it into endpoint latency, where the hits will bury it.

Then go break it deliberately. Point a canary at half production traffic and flush its cache. If the database holds, you have sized the miss path. If it does not, you learned that on a Wednesday morning with a rollback ready instead of at 4pm on a Tuesday.