Your API Isn't Idempotent and You're About to Charge Someone Twice

The Double-Charge Bug Nobody Writes Postmortems About

Your payment endpoint processed the same charge twice. The client's network dropped before it received the response, so it retried. You got two POST /charge requests with identical bodies, treated them as independent, and your Stripe dashboard now shows two successful charges for the same order.

Your API is not idempotent.

Idempotency: applying the same operation multiple times produces the same result as applying it once. GET is idempotent. PUT is idempotent by HTTP definition. POST is not, unless you make it so.

Every distributed system hits this. Retries from clients, from load balancers, from queue consumers. Networks fail. At-most-once delivery is a fiction. You handle duplicate requests deliberately, or you bet that nothing retries and lose that bet.

What Teams Usually Try (And Why It Breaks)

First attempt: deduplication by request body hash. Hash the payload, store it, reject duplicates with 409.

This breaks fast. Two customers legitimately submit identical orders. Two requests arrive with the same hash before either one commits. Race condition: both go through. You add a requestedAt timestamp to the schema, hashes never match across retries, and you're deduplicating nothing.

Second attempt: database unique constraints. UNIQUE on (user_id, amount, created_at). You round timestamps to collapse duplicates and accidentally merge requests that should be separate. The judgment call gets baked into a migration and rots as the schema evolves.

Both approaches put deduplication in the wrong layer.

The Right Layer: Idempotency Keys

Stripe, Braintree, and most payment processors use explicit idempotency keys. The client generates a UUID before making a request and includes it as a header. The server deduplicates on that key.

POST /charges HTTP/1.1
Idempotency-Key: 7f4e3b2a-1c9d-4e8f-b3a2-5d6c7e8f9a0b
Content-Type: application/json

{"amount": 4900, "currency": "usd"}

Server flow:

1. Extract Idempotency-Key 2. Look it up in your key store (Redis or Postgres, whichever has fast point reads) 3. **HIT**: return the stored response, no re-processing 4. **MISS**: process the request, store the result against the key, return the response

Store the **response**, not a boolean flag. Same key gives back the same status code and body. Clients retry without checking whether their previous request succeeded.

The client's second request hits the key store, gets the stored 200, and returns without touching Stripe.

The Race Condition You Will Hit

The naive implementation breaks under concurrency. Two identical requests arrive 5ms apart. Both check the key store simultaneously, both see MISS, both charge the card. You reproduced the original bug inside your deduplication layer.

Fix it with a two-phase DB write:

INSERT INTO idempotency_keys (key, user_id, status, created_at)
VALUES ($1, $2, 'processing', now())
ON CONFLICT (key, user_id) DO NOTHING
RETURNING id

If RETURNING comes back empty, another request already claimed this key. Wait and retry the lookup. If you get a row back, you own this key. When processing finishes, update status to 'complete' and write the serialized response.

This is a pessimistic application-layer lock. Heavier than Redis SET NX, but durable. If the API server crashes mid-request, the 'processing' row persists and the client gets a retryable error it can act on.

Keep It in Middleware

The idempotency check belongs in middleware. Route handlers should not know about deduplication.

Middleware intercepts on the way in (check the key) and wraps on the way out (store before returning). Route handlers stay unaware. You can retrofit this onto existing endpoints without touching any business logic.

Four Decisions You Need to Make

**TTL.** Stripe keeps keys 24 hours. Retries beyond that are new requests with new intent. Longer TTLs cost more storage; set a cleanup job.

**Key scope.** Scope keys per user: (user_id, key) as the unique constraint. A global key namespace lets a client block another user's request by submitting a colliding key string.

**Failed requests.** If your DB write succeeds but the downstream call fails, the key is claimed with an error response stored. On retry, the client gets back the same error. The client needs a new key for a fresh attempt. Put this in your API docs, or you'll debug support tickets about it.

**External services without idempotency support.** If your upstream (email provider, SMS gateway) doesn't accept idempotency keys, you need a "did I already send this?" check in your own records before calling out. That's a business-logic concern and harder to centralize.

Tradeoffs

Idempotency keys solve retry safety but add an extra key store read on every mutation. Redis at p99 might be 1ms. On a high-volume endpoint, that compounds.

The alternative: design operations to be idempotent at the resource level. Let clients own resource IDs. PUT /orders/:client_generated_id is idempotent by HTTP semantics, no extra storage. This falls apart for operations with external side effects. You can't PUT a bank charge or an email send.

Pick idempotency keys for side effects that can't be undone: charges, emails, provisioned resources, SMS. Pick PUT-based resource creation when the concern is your own database state and external calls are either absent or idempotent.

HTTP clients retry on timeout. Load balancers retry on 503. SDK retry budgets fire. If your API has mutations, your clients retry, and you need a deduplication strategy. No strategy means betting on network reliability. The first production incident settles it.