Stop Building Workflow Engines in Your Queue Workers

Stop Building Workflow Engines in Your Queue Workers

The scenario: a user signs up for your SaaS. You need to provision a workspace, send a welcome email, create their first project, and schedule a 3-day follow-up nudge. Simple enough, right?

So you do what seems sensible: a queue job kicks off on signup. One worker provisions, fires a job to the email queue, which triggers a job for project creation, which schedules the follow-up via cron.

You've just built a distributed workflow engine. Out of duct tape.

What Actually Goes Wrong

For a while, it works. Then, at 2am, your email job crashes after the welcome email sends but before it publishes to the next queue. The message gets requeued. The email sends again. The user now has two welcome emails and two starter projects.

You fix it with an idempotency key. The idempotency key has a 24-hour TTL in Redis. A retry happens 25 hours later. The check misses. You're back to square one.

So you add a status column to track where you are:

ALTER TABLE onboarding_jobs ADD COLUMN current_step  VARCHAR(50);
ALTER TABLE onboarding_jobs ADD COLUMN retry_count   INT DEFAULT 0;
ALTER TABLE onboarding_jobs ADD COLUMN last_error    TEXT;
ALTER TABLE onboarding_jobs ADD COLUMN next_retry_at TIMESTAMPTZ;

And a completed_steps JSON blob to avoid re-running steps. And a scheduler to find stalled jobs. Congratulations: you've hand-rolled Temporal, but badly.

The core problem isn't retries. It's that **execution state lives nowhere**. Each worker is stateless; the handoff between steps is a published message that might get lost, duplicated, or processed out of order. There's no single source of truth for "user 42's onboarding is on step 3."

The Better Pattern: Durable Execution

Durable execution flips the model. Instead of coordinating via queues, you write a workflow function that looks like normal imperative code — and the framework makes it resumable.

export async function onboardingWorkflow(userId: string) {
  await provisionWorkspace(userId);
  await sendWelcomeEmail(userId);
  await createStarterProject(userId);
  await workflow.sleep('3 days');
  await sendFollowUpNudge(userId);
}

If the worker running this crashes between step 2 and step 3, the framework replays the history: steps 1 and 2 are already in the event log as completed, so they don't re-run. Execution resumes at step 3. The workflow.sleep('3 days') isn't a cron job — it's a durable timer in the execution log.

This is the core insight: **the event log is the state**. You don't need status columns or retry bookkeeping because the framework already knows exactly where you are.

Tools in this space: **Temporal** (self-hosted or Temporal Cloud, most mature), **Inngest** (serverless, zero infra to run), **Cloudflare Workflows** (if you're already on Workers), and **AWS Step Functions** (for AWS shops who can tolerate JSON state machine definitions).

What Durable Execution Buys You

**Crash recovery without coordination.** When your worker pod dies mid-execution, the next worker replays only the completed steps from the log and continues forward. No manual "find stuck jobs" cron.

**Visibility for free.** Every framework in this space ships a UI that shows which step each workflow instance is on, its full execution history, and any errors. That onboarding_jobs table you were squinting at? Gone.

**Side-effect isolation.** Activities — the individual steps — are where your actual I/O lives. The orchestrator just coordinates. Retrying an activity doesn't re-run your orchestration logic, only the failed step.

**Long-running timers without cron.** The 3-day follow-up is a workflow.sleep() call. Not a separate cron job querying a DB for users who signed up 3 days ago and haven't received a nudge yet.

The Tradeoffs

Durable execution isn't free to operate. Temporal needs a backing store (Postgres or Cassandra) and a separate server cluster. If you're a small team running simple jobs, that overhead isn't worth it — Inngest or Cloudflare Workflows lower the bar considerably, but they add a SaaS dependency you'll need to account for in your availability story.

Workflow code also has a strict constraint: **it must be deterministic**. No Date.now(), no Math.random(), no direct HTTP calls inside the orchestrator. Everything side-effectful goes in an activity. This trips people up, and violating it produces subtle replay errors that are genuinely annoying to debug because the framework silently diverges from the history it expects.

And not everything is a workflow. A single job that does one thing? Plain queue, no question. Fan-out that fires 1000 independent tasks in parallel? Queue with concurrency controls — simpler and cheaper. Durable execution shines specifically for multi-step, stateful, long-running processes where failing mid-stream is expensive to recover from by hand.

The Gut Check

If you have more than two sequential async steps, a manual retry loop, a status column on a jobs table, or a cron job that exists only to find "stuck" records — you're maintaining a workflow engine. You're just doing it the hard way.

The upgrade path isn't dramatic. Extract each queue worker into an activity function. Write an orchestrator that calls them in sequence. Point it at Inngest or Temporal. Delete the status columns.

The result is code that reads like what it does, fails in ways you can see, and recovers without your intervention at 2am.