Every Service Retried Three Times, So One Checkout Click Hit Your Database 81 Times

81 calls for one checkout

I once lost an evening to an outage where a 1.8 second Postgres failover became 40 minutes of downtime, and not a single retry in the stack was wrong on its own.

The chain was boring. Gateway, checkout, orders, pricing. Pricing reads from Postgres. All four services imported the same internal HTTP client, and that client shipped with a default nobody had touched in three years: three attempts, exponential backoff, 200ms base.

Three attempts at four call boundaries is 81 requests to pricing for one click on Place Order. The failover itself took pricing down for under two seconds. The retries kept it down for the rest of the hour, because 81x traffic drained the connection pool, pool exhaustion made healthy requests time out, and timeouts are the input to the retry loop.

The second half of that diagram is the part that still bugs me. The browser gave up after five seconds. Our backoff schedule, summed across hops, ran to roughly forty. So for thirty five seconds the whole chain burned database connections producing a response that no client was holding a socket for. The user, meanwhile, hit refresh twice, which is its own retry policy that I do not control.

Per-service timeouts are not a budget

Every service had a timeout. Checkout allowed 10s, orders 8s, pricing 5s, the pg client 4s. Each number was picked by a different person in a different quarter, each one generous enough to let its own retries finish. Nobody had written down the only number that matters: how much time is left for this request.

That is the actual bug. A timeout says "I will wait this long." A deadline says "this work is worthless after this instant." Those are very different claims, and only the second one composes across a call graph.

gRPC got this right from the start: the caller sends a timeout, each hop subtracts what it has already spent, and the remainder travels downstream. You can do the same over HTTP with one header. The rule is that a hop never starts work it cannot finish, and never waits longer than its caller will.

In Node that is maybe fifteen lines:

// At the edge: one deadline for the whole request tree.
const deadline = Date.now() + 3000;

export async function call(url: string, deadline: number, init: RequestInit = {}) {
  const left = deadline - Date.now() - 50; // 50ms to serialize our own response
  if (left <= 0) throw new DeadlineExceeded(url); // don't even open the socket
  return fetch(url, {
    ...init,
    signal: AbortSignal.timeout(left),
    headers: { ...init.headers, "x-deadline-at": String(deadline) },
  });
}

Push it all the way down. The database is a hop too:

const left = deadline - Date.now() - 50;
await pg.query(`SET LOCAL statement_timeout = ${Math.max(1, left)}`);

That one line deleted a whole class of incident for us. Abandoned queries stop holding locks for 30 seconds after the person who asked has closed the tab.

Retries are a share of traffic, not a property of a call

The other change was giving up on counting attempts per call. The Google SRE book makes the argument better than I will: retries have to be budgeted at the client as a fraction of traffic, because three attempts is harmless at normal error rates and catastrophic at 100% error rates, which is exactly when it fires.

A Finagle style budget, roughly:

class RetryBudget {
  private tokens = 10;
  constructor(private ratio = 0.1, private max = 100) {}

  request() { this.tokens = Math.min(this.max, this.tokens + this.ratio); }

  tryRetry() {
    if (this.tokens < 1) return false;
    this.tokens -= 1;
    return true;
  }
}

Every request deposits 0.1 tokens, every retry spends one, so retries settle at 10% of traffic no matter how bad things get. (The real thing decays tokens on a sliding window so a quiet night does not bank a thundering herd.)

Two more rules came out of the same postmortem. Retry at exactly one layer, the one that owns the idempotency key, and have every other hop pass the failure up untouched. And let an overloaded server say so out loud: 503 with Retry-After, or a header the client knows means do not come back yet. Load shedding only works if the shed load does not immediately return.

What this costs

Deadline propagation is a discipline, not a library. One service that forgets to forward the header silently reopens the hole, and no dashboard will tell you. We ended up emitting a counter for requests arriving without a deadline and alerting when it goes above zero.

Retrying at one layer only means transient blips in deep hops now surface as user visible errors instead of being quietly papered over. That is a genuine loss of success rate on a good day, traded for not falling over on a bad one. I think that trade is obviously correct, but somebody in support will notice and you should tell them first.

Hard deadlines also kill legitimate slow work. A report that honestly needs four seconds dies at three. Do not raise the deadline for it, move it off the request path into a queue, where slow is allowed.

The failover still happens about once a quarter. These days it shows up as a two second latency bump on one graph, and nobody gets paged. Worth reading if you want the math on backoff and jitter: the AWS Builders' Library piece on timeouts and retries and chapter 22 of the Google SRE book.