Stop Doing Real Work in Your Webhook Endpoint

The Mistake

A very normal engineering mistake is turning a webhook endpoint into a tiny monolith.

You start with something innocent:

• verify the signature

• load the tenant

• update the database

• invalidate cache

• call Slack or email

• maybe call another internal API

• return 200

It feels efficient because "the event is right here already."

It is also how you accidentally build a system that behaves badly under exactly the conditions webhooks are supposed to survive: retries, duplicates, bursts, partner outages, and tenant-level traffic spikes.

Providers keep telling us this, pretty explicitly. Stripe says to return a 2xx quickly before complex logic. Slack is even blunter: respond within three seconds, avoid processing in the same process, and use a queue. GitHub gives similar advice.

If your handler is still doing meaningful business work inline, you are fighting the contract.

Why This Design Fails in Production

The first problem is duplicates. Webhook delivery is usually at-least-once, not exactly-once. If your handler sends an email, mutates billing state, and then times out before returning, the provider retries. Now your system gets to rediscover idempotency the expensive way.

The second problem is partial failure. Say the DB write succeeds, cache invalidation fails, and the handler returns 500. Did the event happen or not? Operationally, the answer is "kind of," which is the worst answer.

The third problem is tenancy. One large customer can dump a burst of valid events into the same code path as everyone else. If you process inline, your hottest tenant becomes your platform scheduler.

The fourth problem is rate limits. Internal APIs, email providers, search indexing, and chat notifications all have their own limits. Synchronous handlers turn someone else’s burst into your own downstream outage.

The fifth problem is observability. A single HTTP request span tells you almost nothing once real work fans out across retries, queues, side effects, and replays. If the only metric you have is webhook 500s, you are already late.

A Better Architecture

The better pattern is boring on purpose:

1. Edge receiver validates and normalizes the event. 2. Persist the raw envelope in a durable inbox with an idempotency key and tenant_id. 3. Enqueue lightweight work. 4. Return 202 or 200 immediately. 5. Let background workers perform business logic. 6. Emit downstream changes through an outbox or domain events.

The edge function should be aggressively small. Verify signature, attach tenant context, store the event, enqueue, leave. This is a great fit for edge/serverless runtimes because the job is mostly authentication, normalization, and durable handoff, not long-lived compute.

The important detail is the inbox. Not just a queue. The queue handles delivery. The inbox is your source of truth for "we received event X for tenant Y at time T with payload version V." That record is what lets you dedupe, replay, audit, and migrate consumers later.

A minimal shape looks like this:

await inbox.insert({
  provider: "stripe",
  eventId: event.id,
  tenantId,
  receivedAt: new Date().toISOString(),
  payload,
  status: "queued"
})

await queue.send({ provider: "stripe", eventId: event.id, tenantId })
return new Response("ok", { status: 200 })

That is the boundary. Everything after that is async by design.

What the Workers Should Actually Do

Workers read from the queue, load the inbox record, and execute idempotent business logic. If processing checkout.session.completed should create an internal subscription, then the worker should be able to run twice and still produce one subscription.

This is also where tenancy belongs. Do not let one global FIFO become your fairness policy. Use tenant-aware concurrency and backpressure. A simple version is:

• per-tenant concurrency caps

• per-tenant token bucket rate limits for downstream APIs

• dead-letter queues for poison messages

• explicit retry policy with jitter

That last part matters. Retry behavior should be deliberate, not whatever your SDK happened to ship. Cloudflare Queues documents retries and DLQs clearly; the same principle applies on SQS, Pub/Sub, Kafka consumers, or whatever else you run.

Also: keep cache updates out of the ingress path. Let workers update read models or publish outbox events that invalidate caches. Your webhook receiver is not the right place to care about CDN freshness.

Observability Across the Async Boundary

Most teams instrument the HTTP endpoint and call it a day. That is not enough.

You want to be able to answer:

• Was the event accepted?

• Was it deduped?

• How long did it wait in queue?

• Which tenant is saturating concurrency?

• Which downstream dependency is causing retries?

• Did the worker eventually succeed, fail permanently, or get replayed?

This is where OpenTelemetry messaging conventions help. Treat the queue hop as a first-class trace boundary. Record message IDs, tenant IDs, attempt counts, and links between producer and consumer spans.

The dashboard I want is not "webhook latency." It is:

• inbox write failures

• queue age by tenant

• retry count by event type

• DLQ volume

• worker success latency p50/p95/p99

• downstream 429 and 5xx rates

• replay count after deploy

If you cannot see those, you do not have an eventing system. You have vibes.

Where Durable Execution Fits

If the background work is short and stateless, a queue consumer is enough.

If the process spans hours, waits for approvals, or coordinates multiple external steps, that is where durable execution starts making sense. Cloudflare Workflows and Temporal both exist for a reason: retries, waiting, and state recovery become the product instead of your side project.

The key is not to over-rotate. A webhook receiver should hand off quickly. A worker should do finite idempotent work. A workflow engine should own long-lived coordination. Different boundaries, different failure models.

The Tradeoffs

This architecture is better, not free.

• You accept eventual consistency instead of synchronous completion.

• You now operate an inbox, queue, worker fleet, and replay tooling.

• Schema evolution gets real because events live longer.

• Product teams need to understand that 200 from the provider does not mean the user-visible effect is finished.

You should also roll this out behind flags. Route one tenant, then one event type, then the rest. Event systems are easy to build and embarrassingly easy to migrate badly.

The Rule I Use

If an external system is pushing events into your platform, your first job is durable acceptance, not immediate business completion.

That one architectural shift fixes a surprising amount: retries become normal, tenant spikes become containable, rate limits become schedulable, caches become downstream concerns, and observability finally matches reality.

Do less in the webhook. Build a better boundary instead.