You Saved the Idempotency Key After the Charge, So the Retry Charged Again

The ticket that made no sense

Three customers paid twice in one night. Same amount, two charges seconds apart, and both requests carried the same Idempotency-Key header. That last part is what I couldn't explain, because the endpoint checked the key. I wrote the check. It was about eleven lines and it looked like this.

app.post("/payments", async (req, res) => {
  const key = req.header("Idempotency-Key");

  const saved = await db.oneOrNone(
    "select response from idem_key where key = $1", [key]
  );
  if (saved) return res.json(saved.response);

  const charge = await psp.charge({
    amount: req.body.amount,
    source: req.body.source,
  });

  await db.none(
    "insert into idem_key (key, response) values ($1, $2)", [key, charge]
  );
  res.json(charge);
});

Read the key, replay the saved response if there is one, otherwise charge the card and save. Everyone on that PR, me included, read it as if those two database calls were one operation. They are not. There is a gap between the select and the insert, and the gap is exactly as long as the card processor takes to answer.

Our mobile client used a 10 second HTTP timeout. That night the processor was taking about 11.

The retry lands while the first request is still sitting inside psp.charge. Nothing has been written yet, so the select honestly finds nothing, and the second request does precisely what it was told to do. Two charges, one key, and not a single line of code that is wrong on its own.

You cannot fix this by checking more carefully. Check-then-act has a window by construction, and a select ... for update on a row that does not exist locks nothing at all. What has to change is the meaning of the table. I was using it as a log of work that finished. It needs to be a claim on work that is about to start.

Write the key before you do the work

Insert first and let a unique constraint decide who wins.

create table idem_key (
  tenant_id    uuid        not null,
  endpoint     text        not null,
  key          text        not null,
  fingerprint  text        not null,
  status       text        not null
               check (status in ('in_flight', 'committed', 'failed')),
  response     jsonb,
  provider_ref text,
  locked_until timestamptz not null,
  created_at   timestamptz not null default now(),
  primary key (tenant_id, endpoint, key)
);

The key is scoped per tenant and per endpoint, which matters more than it sounds like it does. A global key namespace lets one tenant's UUID collide with another's, and lets a client reuse a refund key on a payment route.

const fingerprint = sha256(canonicalJson(req.body));

const claim = await db.oneOrNone(
  `insert into idem_key
     (tenant_id, endpoint, key, fingerprint, status, locked_until)
   values ($1, '/payments', $2, $3, 'in_flight', now() + interval '90 seconds')
   on conflict do nothing
   returning *`,
  [tenantId, key, fingerprint]
);

if (!claim) {
  const row = await loadKey(tenantId, "/payments", key);
  if (row.fingerprint !== fingerprint) {
    return res.status(422).json({ error: "key reused with a different body" });
  }
  if (row.status === "in_flight") {
    return res.status(409).set("Retry-After", "2").json({ error: "in progress" });
  }
  return res.status(row.status === "committed" ? 200 : 402).json(row.response);
}

The insert is the lock. Whoever wins it owns the charge, and every concurrent twin gets told what the owner is doing instead of guessing. The fingerprint catches the other common client bug, which is the same key sent with a different amount. Replaying the first response there would be worse than failing, so that one gets a 422.

Then the part that usually gets skipped: hand your key to the processor as well.

const charge = await psp.charge(body, { idempotencyKey: `${tenantId}:${key}` });

Your transaction and the processor's ledger are separate systems and they will never commit together. So there is still a moment where the charge exists and your row says in_flight. You cannot remove that moment, but you can make it recoverable rather than invisible.

A sweeper picks up rows whose locked_until has passed and asks the processor what happened under that key. Because the key went downstream as their idempotency key too, the answer is authoritative instead of a guess: either a charge exists under it or it doesn't. The row settles, and the next retry gets a real response. Ours has resolved 23 rows in eight months, nearly all of them from deploys that cycled pods mid request.

What it costs

• One extra insert on the hot path. Cheap, and contention is scoped to a single key rather than a shared row or table.

• A retention decision you now own. We keep rows 72 hours and drop them nightly. Stripe keeps 24. A longer window is kinder to a phone that retries after a day offline, and it also means a stale key can resurrect a cart the user walked away from.

• 409 rather than waiting. Holding the connection open until the first attempt lands feels friendlier, but it just relocates the timeout into your proxy and ties up a worker. Make the client come back.

• The client has to own the key. If your server mints one per request, none of this does anything. That was our actual root cause for two releases: the SDK generated a fresh UUID inside its own retry loop, so the server logic was correct and the double charges kept arriving anyway.

• Failures need a verdict. A decline is a real outcome, so store it as failed with the body and replay it. A 503 from the processor is not an outcome, so release the claim and let a genuine retry through.

If you change one thing, move the insert above the side effect. Fingerprints, sweepers, and TTLs are all cleanup around that single reordering.