Back to BlogNode.js

A Deploy Restarted the Worker Mid-Charge, and Your status Column Billed Nine Customers Twice

A single status column on the orders table cannot survive a pod restart in the middle of a payment call, because the effect and the record of the effect are two writes to two systems. Here is the failure, the append-only step journal that replaces it, and the versioning tax you take on in exchange.

durable executionidempotencystate machinespostgresworkers
A Deploy Restarted the Worker Mid-Charge, and Your status Column Billed Nine Customers Twice

A status column on the orders table. Nine values, one handler per value, a cron every thirty seconds that pulls rows and advances them. It ran for fourteen months and nobody touched it, which is the dangerous kind of working.

Then a Tuesday afternoon deploy rolled the pods. Kubernetes sent SIGTERM, a handler was somewhere inside a payment call, and nine orders came out the other side charged twice. Support found it before our alerts did, which tells you something about the alerts.

The two ways to lose

The bug has nothing to do with the payment provider.

The charge happens over there. The record that the charge happened lives in your database. Those are two writes to two systems and you cannot make them atomic, so you get to pick which failure you want:

  • Write charging first, then call the provider. A crash after the call leaves the row in charging, the charging handler grabs it on the next tick, and you charge again.

  • Call the provider first, then write charged. A crash leaves the row in pending, and you charge again.

Most teams find this and reach for a timestamp. Set charge_started_at, skip rows younger than two minutes. That narrows the window instead of closing it, and now a genuinely slow payment looks exactly like a dead worker. Ask me how I know.

The column answers one question: where is this row in my pipeline. It cannot answer the question that actually matters during recovery, which is what has already happened outside my database. And a single status usually covers several effects. Ours hid a charge, a shipping label, a confirmation email, and a ledger row all under charging. Resuming that status means replaying all four, whether or not any of them went through.

Store the steps, not the state

The shape that works is boring. Two tables.

A run row holds the intent and the input, written once and never updated: order 8812, charge 4,200 rupees, ship to this address. Then an append-only run_step table, one row per step per run, with a unique constraint on (run_id, name). The step row carries its own state, the attempt count, and the provider's response as JSON.

The executor is a loop with no memory of its own. Read the journal, find the first step that is not recorded as done, run it, append the result. Crash anywhere and the next worker reads the same journal and reaches the same conclusion.

async function step<T>(run: Run, name: string, fn: (key: string) => Promise<T>): Promise<T> {
  const prior = await db.runStep.findUnique({ where: { runId_name: { runId: run.id, name } } });
  if (prior?.state === "done") return prior.result as T;

  const key = `${run.id}:${name}`; // stable across every retry, forever
  await db.runStep.upsert({
    where: { runId_name: { runId: run.id, name } },
    create: { runId: run.id, name, state: "running", attempts: 1 },
    update: { state: "running", attempts: { increment: 1 } },
  });

  const result = await fn(key);
  await db.runStep.update({
    where: { runId_name: { runId: run.id, name } },
    data: { state: "done", result: result as object },
  });
  return result;
}

And the flow reads like a function again, because it is one:

const charge = await step(run, "charge", (key) =>
  payments.charge({ amount: run.input.amount, idempotencyKey: key }));

const label = await step(run, "label", (key) =>
  carrier.buyLabel({ orderId: run.input.orderId, requestId: key }));

await step(run, "notify", (key) =>
  mail.send({ to: run.input.email, template: "shipped", dedupeKey: key, tracking: label.code }));

What replay actually buys you

Not exactly-once. I want to be blunt about that, because the marketing around durable execution keeps implying otherwise.

Look at step() again. fn() can succeed and the state: "done" write can still fail. That window did not go away, it moved. What changed is that the retry now arrives at the provider carrying the same idempotencyKey it used the first time, so the provider recognises it and hands back the original charge instead of creating a second one. You traded exactly-once execution, which nobody has, for at-least-once execution plus a key stable enough that the other side can deduplicate.

Which means the key is the actual design work. ${run.id}:${name} survives restarts, redeploys, and a human clicking retry three weeks later. crypto.randomUUID() inside the handler survives nothing. Neither does anything derived from Date.now(), and that is the discipline this architecture leaks into code review: the orchestration body gets no clocks, no randomness, no reading of mutable config. If a step needs the current time, make the time a step and let its result land in the journal, where the replay can read it back.

What it costs

The honest bill, in rough order of how much it hurt us.

Versioning in-flight runs is the real tax. Inserting a step between charge and label is free for new runs and a problem for the forty thousand runs already past charge. You end up pinning a flow_version on the run row and keeping the old branch alive until the stragglers drain, and then someone has to actually go delete it.

Operations gets harder before it gets easier. Nobody can unstick an order with an UPDATE orders SET status = 'charged' anymore, which is the point, but it means you owe your support engineers a real tool: show the run, skip a step with a recorded reason, force a retry.

The journal grows. One row per step per run adds up faster than people expect, so decide retention and partitioning on day one rather than during the incident where run_step is your largest table.

And you should genuinely consider not building it. Temporal, Restate, and DBOS hand you replay, durable timers, and a UI that already answers "where is order 8812". The counterweight is that you have adopted an engine with its own upgrade path and its own failure modes, and local development gets heavier. We had four flows and a Postgres we already trusted, so about two hundred lines of executor won. At fifteen flows with human approval steps and week-long timers I would buy instead.

Also, sometimes the status column is correct. Two states, idempotent steps, nothing involving money or anything a customer can see twice. Don't build a journal to track whether you generated a thumbnail.

Seeing a stuck run

The payoff nobody mentions in the design doc is the queries you suddenly get for free. Steps in running for longer than their timeout, which is your real stuck-work alert. Steps with attempts > 3, which is where a provider outage shows up first. Oldest incomplete step per flow, which is the one number I would put on a dashboard if I only got one.

The status column was never wrong, exactly. It was a derived value that we had promoted to source of truth. If your team misses it, give it back to them as a view over the journal.