Your Hot Cache Key Expired Under Load, and 2,000 Concurrent Queries Just Flattened Your Database

The 60-Second TTL That Took Down Black Friday

A 60-second TTL looks innocent in code review. You have an expensive endpoint—say, the homepage catalog or current exchange rates—that takes 80ms to build from Postgres. You slap a standard cache-aside check around it: check Redis, return on hit, query Postgres on miss, and write the value back with a 60-second expiration.

In staging, it flies. In production under 50 requests a second, it flies.

Then traffic climbs to 2,500 requests a second. For fifty-nine seconds, Redis handles the load without breaking a sweat. Your database CPU sits at 4%. Then second 60 arrives. Redis evicts the key.

In the next 80 milliseconds—the time it takes Postgres to execute that single query—200 new requests land on your API pods. Every single one checks Redis. Every single one gets a cache miss. None of them know that another request is already fetching the exact same data.

All 200 requests fire the exact same expensive SQL query at Postgres at the exact same instant. This is a cache stampede, also known as the thundering herd or dogpile effect.

Anatomy of a Connection Pool Meltdown

What happens next is not a gradual slowdown. It is an immediate cliff.

Postgres does not run queries in an unbounded ether. Your application uses a connection pool, usually sized between 20 and 50 connections per pod to keep database memory bounded. When 200 queries land simultaneously, the pool saturates in milliseconds.

The remaining 150 queries sit in driver queues waiting for an available socket. Under that queue depth, query execution times stretch from 80ms to three seconds. Context switching and lock contention push database CPU to 100%.

While those 200 queries crawl, another 2,500 requests arrive every second. Thousands of open HTTP connections wait on stalled promises. Event loops starve. Memory climbs.

Then Kubernetes health checks ping /healthz. Because the health check cannot acquire a database connection, the pod fails its liveness probe. Kubernetes terminates the pod and routes incoming traffic to the surviving instances. Those pods take double the traffic, hit the same cache miss, and collapse within seconds.

Why Distributed Mutex Locks Make It Worse

The instinctive fix is to put a distributed lock around the cache refill. If the key is missing, acquire a Redis lock with SET lock:catalog NX EX 5. The winner queries the database; everyone else waits.

async function getWithLock(): Promise<Data> {
  const cached = await redis.get("catalog");
  if (cached) return JSON.parse(cached);

  const locked = await redis.set("lock:catalog", "1", "NX", "EX", 5);
  if (!locked) {
    await sleep(50);
    return getWithLock();
  }

  try {
    const data = await db.query("SELECT * FROM products");
    await redis.set("catalog", JSON.stringify(data), "EX", 60);
    return data;
  } finally {
    await redis.del("lock:catalog");
  }
}

In high-throughput systems, this pattern replaces one failure mode with two:

• Spin-lock storms on Redis. If 2,400 requests fail to get the lock, they sleep for 50ms and hammer Redis again. You turn a database problem into a Redis CPU spike of 50,000 commands a second.

• Latency cliffs for users. The 2,399 requests that missed the lock sit in sleep loops. If the winning query hits a slow plan, thousands of users stare at spinning wheels until their browsers time out.

Distributed locks are the wrong abstraction for local query deduplication. You do not need cluster-wide consensus across fifty pods just to stop one Node.js process from asking Postgres for the same row twenty times.

Fix 1: In-Process Request Coalescing (Singleflight)

Most of your duplicate queries originate inside the same application process. If one instance handles 300 requests a second, 30 of them hit the cache miss concurrently. You can collapse those 30 requests into one database call right inside process memory.

Go calls this pattern singleflight. In JavaScript, you can implement it in twenty lines using a map of in-flight promises.

Here is a clean implementation in TypeScript:

export class Singleflight {
  private inFlight = new Map<string, Promise<any>>();

  async do<T>(key: string, fn: () => Promise<T>): Promise<T> {
    const active = this.inFlight.get(key);
    if (active) return active as Promise<T>;

    const promise = fn().finally(() => {
      this.inFlight.delete(key);
    });

    this.inFlight.set(key, promise);
    return promise;
  }
}

When 50 requests arrive on Pod A during an 80ms window, the first caller executes fn(). Callers 2 through 50 receive the exact same promise instance and await it. When the query resolves, all 50 callers receive the result simultaneously.

Across ten pods, Postgres receives ten queries instead of five hundred. Ten queries run comfortably inside any connection pool without queuing.

Fix 2: Probabilistic Early Expiration (XFetch)

Singleflight stops the stampede when a key expires, but callers still wait 80ms while the key refreshes. If you want zero cache-miss latency, you cannot wait for the key to die before rebuilding it.

Background cron jobs work for five static keys. They do not work when you cache hundreds of thousands of tenant-specific dashboards or personalized feeds.

The mathematically sound solution is probabilistic early expiration, known as the XFetch algorithm. Instead of letting keys expire abruptly, reads calculate an early refresh probability based on how long the computation takes.

As expiration approaches, incoming reads evaluate this condition:

-delta * beta * Math.log(Math.random()) > ttl_remaining

Here, delta is the time in seconds it took to run the query, beta is an aggressiveness multiplier (default 1.0), and Math.random() produces a float between 0 and 1. Because the natural logarithm of a number close to 0 is a large negative number, -Math.log(random) yields an exponential distribution.

When the key has 50 seconds left on a 60-second TTL, the chance of triggering a refresh is practically zero. With 500ms left, the probability spikes. Exactly one read triggers the database refresh early in the background, updates Redis, and resets the window.

interface CacheItem<T> {
  value: T;
  delta: number;
  expiry: number;
}

async function getWithXFetch<T>(
  key: string,
  ttl: number,
  fetcher: () => Promise<T>,
  beta = 1.0
): Promise<T> {
  const raw = await redis.get(key);
  const now = Date.now() / 1000;

  if (raw) {
    const item: CacheItem<T> = JSON.parse(raw);
    const shouldRefresh =
      -item.delta * beta * Math.log(Math.random()) > item.expiry - now;

    if (!shouldRefresh) return item.value;
    void refresh(key, ttl, fetcher);
    return item.value;
  }
  return refresh(key, ttl, fetcher);
}

async function refresh<T>(
  key: string,
  ttl: number,
  fetcher: () => Promise<T>
): Promise<T> {
  const start = performance.now();
  const value = await fetcher();
  const delta = (performance.now() - start) / 1000;
  const expiry = Date.now() / 1000 + ttl;

  await redis.set(key, JSON.stringify({ value, delta, expiry }), "EX", ttl * 2);
  return value;
}

Notice that Redis TTL is set to twice the logical expiry (ttl * 2). This ensures that even if early refresh triggers late, the old value remains available to serve readers while the background task completes.

Tradeoffs and Operational Realities

Every caching strategy trades memory, complexity, or freshness. Choosing between singleflight and XFetch depends on how your traffic behaves.

• Singleflight requires zero metadata overhead. You do not store execution times or envelope objects in Redis. However, singleflight operates per process. If you run 200 Kubernetes pods, a cold key miss will still generate 200 database queries. That is vastly better than 20,000 queries, but under large cluster scales, 200 heavy queries can still cause a brief connection pool flutter.

• XFetch eliminates latency spikes entirely by keeping hot keys warm. The tradeoff is storage overhead and background load. Every cached object needs an envelope tracking delta and expiry. If your data changes infrequently, refreshing early based on probabilistic timers burns occasional database cycles on data nobody requested yet.

• Memory leak hazards with promise maps. In singleflight, if your database query hangs indefinitely because of an unconfigured socket timeout, that promise stays in the inFlight map forever. Always wrap the database call in an explicit AbortSignal with a strict deadline.

The most resilient production setups layer both techniques. Use XFetch on hot, latency-sensitive read paths so users never experience a cache miss. Back every database fetcher with a local singleflight group so that when cold keys do expire, your database connection pool never takes more than one hit per instance.