Your Distributed Rate Limiter Synced Every Request to Redis, and a 4ms Network Spike Cascaded Your Whole Gateway

The Incident: 15 Milliseconds of Jitter, 100% Outage

At 2:15 PM on a Tuesday, ingress latency jumped from 2 milliseconds to six seconds. Nothing had deployed. CPU usage across our twenty-four API gateway instances sat at a comfortable 14%. But within two minutes, upstream clients were seeing cascades of 504 Gateway Timeout errors, Kubernetes liveness probes began failing across the gateway fleet, and pods were being killed and restarted in a thundering-herd panic.

The culprit was not an exploit or a memory leak. It was a distributed rate limiter doing exactly what it was coded to do.

Every single HTTP request hitting the gateway executed a synchronous Redis Lua script to enforce a sliding-window rate limit before routing downstream. Redis ran a routine background persistence snapshot—a BGSAVE. The OS copy-on-write page table allocation took 15 milliseconds on a 32GB memory footprint. During those 15 milliseconds, the single-threaded Redis event loop stopped serving commands.

Because 35,000 requests per second were entering the gateway, 500 requests backed up across the gateway fleet every millisecond. Node and Go worker pools saturated their socket backlogs, connection pools ran dry, and the gateway stopped accepting TCP handshakes.

The rate limiter had been installed to protect backend databases from collapsing under load. Instead, it became the exact fuse that blew the entire platform.

The Trap of the Synchronous Sliding Window

Most engineering teams start with an off-the-shelf rate limiter tutorial. The standard recipe uses Redis with either a sliding-window log via sorted sets (ZADD, ZREMRANGEBYSCORE, ZCARD) or a sliding-window counter inside an atomic Lua script:

// The intuitive, fatal pattern: blocking network I/O on every request
export async function rateLimitMiddleware(req: Request, res: Response, next: NextFunction) {
  const key = `rate:${req.headers['x-api-key'] || req.ip}`;
  const now = Date.now();
  const windowMs = 60_000;
  const limit = 1000;

  try {
    // Synchronous network hop directly in the request path
    const allowed = await redis.evalsha(RATE_LIMIT_SHA, 1, key, now, windowMs, limit);
    if (!allowed) {
      return res.status(429).json({ error: 'Too Many Requests' });
    }
  } catch (err) {
    // The dilemma: fail-open invites DDoS, fail-closed drops paying traffic
    logger.error({ err }, 'Redis rate limit check failed');
    return res.status(503).json({ error: 'Service Unavailable' });
  }

  next();
}

This pattern holds up during staging tests and modest traffic. But at production scale, placing a remote data store on the synchronous path of every request creates three fatal failure modes:

• **Tail Latency Amplification:** If your API handles 40,000 requests per second, you are forcing 40,000 round-trips per second onto Redis. The P99 latency of Redis becomes the hard latency floor of every single route in your system.

• **Connection Pool Exhaustion:** If Redis encounters even a micro-stall—packet loss, thread scheduling jitter, or memory defragmentation—inbound HTTP requests stall waiting for a response. Web servers run out of file descriptors, and worker queues fill to capacity.

• **The Binary Failure Dilemma:** When Redis degrades, your error handler faces an impossible choice. If you fail closed, a minor cache slowdown knocks out 100% of your legitimate users. If you fail open blindly, a malicious crawler will punch straight through to your unbuffered Postgres database.

The Better Architecture: Local Token Buckets with Asynchronous Leases

High-throughput systems at Stripe and Envoy do not ask a central database for permission on every HTTP request. Instead, they decouple enforcement from synchronization using a two-tier model: an in-memory local token bucket paired with asynchronous batched leases.

Each gateway instance evaluates rate limits against a local in-memory bucket. Checking and decrementing an in-memory counter takes less than 100 nanoseconds and requires zero network I/O.

To keep global state synchronized across dozens of gateway instances, workers acquire tokens from Redis in batches. Instead of deducting one token per request, a gateway instance requests a lease of tokens (for example, 50 tokens or a 250-millisecond quota slice) from the central Redis coordinator. Incoming requests consume from this local reservoir.

When the local reservoir drops below a low-water mark (say, 20% remaining), a background worker asynchronously asks Redis for another batch. If the gateway suddenly sees zero traffic, the local lease simply expires.

// Two-tier token bucket: zero network I/O in the request lifecycle
class LocalTokenLease {
  private localTokens: number = 0;
  private isRefilling: boolean = false;

  constructor(
    private readonly tenantId: string,
    private readonly batchSize: number = 50
  ) {}

  public tryConsume(): boolean {
    if (this.localTokens > 0) {
      this.localTokens--;
      if (this.localTokens < this.batchSize * 0.2 && !this.isRefilling) {
        this.triggerAsyncRefill();
      }
      return true;
    }

    // Bucket empty: reject immediately or trigger emergency sync
    return false;
  }

  private async triggerAsyncRefill(): Promise<void> {
    this.isRefilling = true;
    try {
      const granted = await redis.evalsha(LEASE_SCRIPT_SHA, 1, this.tenantId, this.batchSize);
      this.localTokens += Number(granted);
    } catch (err) {
      logger.warn({ err, tenant: this.tenantId }, 'Failed to refill token lease');
    } finally {
      this.isRefilling = false;
    }
  }
}

By decoupling request evaluation from the network, a Redis hiccup cannot stall incoming HTTP traffic. The fast path stays fast.

Handling Panic Mode and Redis Outages

What happens when Redis crashes or drops off the network entirely?

Instead of choosing between blind fail-open and blind fail-closed, wrap the synchronization tier in a circuit breaker with a dedicated fallback: Panic Mode.

If the error rate or latency of Redis lease requests exceeds a strict threshold (for example, P95 above 10ms for three consecutive seconds), trip the circuit. In Panic Mode:

• The gateway stops attempting to reach Redis, preventing connection storms and freeing socket resources.

• Rate limits switch to static, conservative local fallbacks. For instance, each gateway pod enforces a hard ceiling of 30 requests per second per IP directly in memory.

• Low-priority read endpoints are throttled first (load shedding), preserving downstream capacity for critical business mutations like payment charges.

Legitimate users continue using the API unimpeded. Attackers attempting to flood the system during the outage are still constrained by the local in-memory caps. Once Redis returns to healthy response times, the circuit resets and batched leases resume.

Operational Realities and Trade-Offs

No architectural change comes without trade-offs. Moving to batched token leases introduces real operational compromises you have to accept:

• **Enforcement Drift:** If a client distributes requests across thirty gateway instances simultaneously, each instance consumes its local allotment before syncing. A client with a limit of 1,000 requests per minute might slip 1,050 requests through during an aggressive burst. For 99% of APIs, this slight fuzziness is completely irrelevant. Rate limiting is an abuse mitigation system, not financial ledger accounting.

• **Gateway Memory Limits:** You cannot keep a token bucket in memory for every client IP that ever pinged your site. You need a bounded, fixed-capacity LRU cache with an aggressive TTL (e.g., 2 minutes of inactivity). Unseen clients start with an empty local bucket that fetches an initial seed lease.

• **Endpoint Tiering:** Treat high-risk endpoints differently. A search autocomplete endpoint can use generous batched leases with relaxed drift tolerances. A bank transfer or checkout route can bypass batching and use an atomic, synchronous check with a strict 5ms timeout and an idempotency key.

Putting a central database on the critical path of every ingress packet creates an outage waiting for a latency spike. Move enforcement into memory, batch your remote synchronization, and give your gateway the autonomy to survive when the cache stumbles.