You Published to Kafka Right After db.commit(), and a Pod Crash Lost Three Thousand Invoices

The Illusion of the Sequential Write

Every backend engineer has written this function at least once:

async function createInvoice(req: Request) {
  const invoice = await db.transaction(async (tx) => {
    return await tx.invoices.create(req.body);
  });

  await kafkaProducer.send({
    topic: 'invoices.created',
    messages: [{ key: invoice.id, value: JSON.stringify(invoice) }],
  });

  return invoice;
}

It looks clean, passes your local integration suite in 12 milliseconds, and works smoothly across staging. In production, under real operational pressure, this snippet is an active data corruption incident waiting to trigger.

This code assumes that the database transaction and the message emission belong to the same execution boundary. They do not. You are attempting a dual write across two completely disconnected, distributed systems without a coordinator.

Look at the failure window between lines 3 and 5. The PostgreSQL transaction commits. The invoice row exists durably on disk. Immediately after, before the HTTP connection to your Kafka broker finishes:

• Kubernetes kills the pod due to an unrelated node rebalance or OOM spike.

• A transient network blip severs the connection to the Kafka broker.

• The process receives a SIGKILL during a rolling deployment.

The event vanishes into the ether. Postgres recorded the invoice, but your billing worker, search indexer, and customer notification service will never know it exists. Your read models and downstream services are now permanently out of sync with your primary database.

Swapping the order makes the failure mode worse. If you publish to Kafka first and commit to Postgres second, Kafka immediately delivers the invoices.created message to downstream consumers. Those consumers query Postgres for the invoice and get a 404 because your API's database transaction hasn't committed yet. If the database transaction subsequently fails on a unique constraint or deadlock, you have broadcasted a phantom event for a record that never existed.

The Distributed Transaction Mirage

When engineers recognize the dual-write flaw, the instinct is often to look for distributed transactions. Decades of enterprise software literature suggest Two-Phase Commit (2PC) or XA transactions.

In modern cloud services, 2PC is a non-starter. Neither Apache Kafka nor AWS SQS supports joining an XA transaction with PostgreSQL. Even if they did, 2PC is a blocking protocol. If the coordinator crashes during the prepare phase, participating nodes must hold row locks and connection resources indefinitely until the coordinator recovers. In a high-throughput microservice, holding database locks across network-bound coordination collapses throughput and drags your availability down to the product of every participating dependency's uptime.

Distributed systems do not allow you to cheat the CAP theorem with wishful thinking. If you want atomicity between your persistent state and your outgoing event stream, both writes must happen inside a single storage engine.

The Fix: The Transactional Outbox

The standard solution is the Transactional Outbox pattern. Instead of talking to Kafka directly from the request handler, write the outbound message to an outbox table in the *same* PostgreSQL transaction that stores your business record:

CREATE TABLE outbox_events (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  aggregate_type VARCHAR(64) NOT NULL,
  aggregate_id VARCHAR(64) NOT NULL,
  event_type VARCHAR(64) NOT NULL,
  payload JSONB NOT NULL,
  created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp()
);

Your application handler simplifies to a single local transaction:

await db.transaction(async (tx) => {
  const invoice = await tx.invoices.create(req.body);

  await tx.outboxEvents.create({
    aggregateType: 'Invoice',
    aggregateId: invoice.id,
    eventType: 'InvoiceCreated',
    payload: {
      id: invoice.id,
      amount: invoice.amount,
      customerId: invoice.customerId,
      createdAt: invoice.createdAt,
    },
  });

  return invoice;
});

Because both writes execute within the same ACID transaction, atomicity is guaranteed by Postgres. Either both the invoice and the outbox event exist on disk, or neither does. No ghost events. No dropped records.

Moving Bits: CDC vs Polling SKIP LOCKED

Once the event sits in Postgres, you need an asynchronous relay to move it to Kafka. You have two viable strategies.

The first approach is a polling worker. You run a background loop that reads pending rows using SELECT ... FOR UPDATE SKIP LOCKED, pushes them to Kafka, and deletes or marks them sent. This works adequately at low volume (under 50 events per second). Beyond that, polling taxes the primary database with continuous index scans and generates high write amplification from updating row flags.

The industry-standard approach is Change Data Capture (CDC) via PostgreSQL logical decoding. Tools like Debezium connect as a replication client directly to PostgreSQL's Write-Ahead Log (WAL). When your transaction commits, Postgres writes the change to the WAL. Debezium reads the WAL stream asynchronously and routes the payload to Kafka.

With the Debezium Outbox Event Router transform (io.debezium.transforms.outbox.EventRouter), Debezium inspects the aggregate_type to choose the destination topic, sets the Kafka message key to aggregate_id (ensuring per-entity partition ordering), and strips out internal CDC metadata before publishing.

The Tradeoffs Nobody Mentions in Architecture Diagrams

Implementing the Transactional Outbox pattern eliminates silent data loss, but it introduces three distinct operational burdens.

1. Replication Slot Lag Will Fill Your Disk

PostgreSQL logical replication uses replication slots to track consumer positions. PostgreSQL guarantees that WAL segments will not be deleted until all active replication slots have processed them.

If your Debezium connector crashes, loses network access to Kafka, or falls behind during a heavy load spike, PostgreSQL will refuse to purge WAL files. The pg_wal directory will grow continuously until your database disk fills up and PostgreSQL shuts down into read-only crash recovery.

Mitigate this by setting max_slot_wal_keep_size in postgresql.conf (for example, to 64GB). If a slot lags beyond that threshold, Postgres invalidates the slot rather than sacrificing the database. Pair this with a high-priority PagerDuty alert on pg_replication_slots.confirmed_flush_lsn lag.

2. At-Least-Once Delivery Forces Idempotency

Log tailing provides at-least-once delivery, not exactly-once. If Debezium publishes an event to Kafka but crashes before committing its source offset back to Postgres or Kafka Connect's offset topic, it restarts and replays those WAL records.

Downstream consumers will receive duplicates. Every consumer subscribing to your outbox events must be strictly idempotent. Track incoming message UUIDs in a dedicated deduplication table:

INSERT INTO processed_events (event_id, processed_at)
VALUES ($1, NOW())
ON CONFLICT (event_id) DO NOTHING;

If zero rows were inserted, discard the message immediately.

3. Outbox Table Bloat

If your system processes two million events a day, your outbox_events table accumulates two million inserts. If you delete them with a naive cleanup query (DELETE FROM outbox_events WHERE created_at < NOW() - INTERVAL '1 day'), you will generate millions of dead tuples, bloat your table indices, and trigger aggressive autovacuum locks.

Instead, partition your outbox table by day (PARTITION BY RANGE (created_at)). Dropping an entire partition with DROP TABLE outbox_events_2026_10_20 is a metadata-only operation that finishes in two milliseconds without creating vacuum churn.

The Operational Checklist

• Never call message broker SDKs (kafka.send(), sqs.sendMessage()) inside or directly after an un-outboxed HTTP request handler.

• Write business state and outbound event payloads within the same local database transaction.

• Prefer WAL-based Change Data Capture over SKIP LOCKED polling when event throughput exceeds 100 writes per second.

• Configure max_slot_wal_keep_size to prevent a stalled replication slot from consuming 100% of your disk.

• Enforce unique event IDs and consumer-side deduplication tables across all downstream listeners.