Your Distributed Lock Expired During a GC Pause, and Two Workers Just Corrupted Your Storage

The Ten-Second Lock That Wasn't

Most distributed locking code looks clean, passes code review, and works fine in local testing. You spin up Redis, generate a UUID, and set a key with an expiration:

const lockKey = `lock:settlement:${batchId}`;
const acquired = await redis.set(lockKey, workerId, "NX", "PX", 10000);

if (!acquired) {
  return; // Another worker has this batch
}

try {
  const result = await processSettlementBatch(batchId);
  await storage.saveBatchSummary(batchId, result);
} finally {
  await redis.eval(RELEASE_LOCK_SCRIPT, 1, lockKey, workerId);
}

The logic feels airtight. Redis ensures only one caller gets the key with NX. The 10-second TTL prevents deadlocks if the worker crashes before reaching finally. The Lua script guarantees the worker only deletes its own lock.

Then you run this in production under high memory pressure.

Worker A acquires the lock. It pulls 40,000 transaction rows and begins transforming them in memory. Right before calling saveBatchSummary, the runtime hits a major garbage collection pause. Or the VM gets descheduled by the cloud hypervisor. Or a Kubernetes node runs out of burstable CPU credits on a shared host.

The process stops executing user code for 12 seconds.

While Worker A is frozen, the wall clock keeps running. Redis reaches the 10-second mark and drops the key. Worker B enters the queue, asks for lock:settlement:123, and gets it. Worker B processes the batch, writes its report to storage, and cleans up.

A split-second later, Worker A unfreezes. As far as Worker A's thread knows, zero time passed. It proceeds directly to storage.saveBatchSummary(...) and writes its stale payload, silently overwriting the fresh data Worker B just saved.

The mutual exclusion guarantee completely collapsed. You paid the latency penalty of a distributed lock, yet still ended up with a concurrent write race.

The Watchdog Fallacy

When teams hit this failure mode, the immediate reaction is usually: "Let's increase the TTL to 60 seconds," or "Let's add a background heartbeat to refresh the lease."

Redisson and similar libraries implement this pattern as a lock watchdog. A background thread runs every few seconds and sends PEXPIRE to bump the lock's expiration as long as the worker is alive.

It sounds comforting, but it does not fix the root failure.

If Worker A stalls because of a full-process pause—like a JVM stop-the-world GC run, a Node.js event loop starved by synchronous compression, or an OS paging storm—the watchdog thread pauses alongside the worker thread. It cannot send renewal heartbeats because the entire process has stopped executing instructions.

Redis will still expire the lease. Worker B will still acquire the lock. And Worker A will still wake up with an expired lease and issue its write.

Martin Kleppmann detailed this breakdown in his critique of how to do distributed locking, analyzing why algorithms like Redlock fail without stronger ordering guarantees. While Redis author Salvatore Sanfilippo defended the practical ergonomics of time-based locks in his response, the theoretical reality remains unyielding: in an asynchronous distributed system, physical clocks and timed leases cannot guarantee mutual exclusion.

A worker cannot safely assume it still owns a lock simply because it checked the lease earlier in its execution path.

The Better Architecture: Monotonic Fencing Tokens

To stop a delayed worker from corrupting data, the lock coordinator cannot act alone. The storage layer must actively defend itself against out-of-order writes.

This is where fencing tokens come in.

Every time the lock coordinator hands out a lock, it attaches a strictly monotonic counter—an integer that increments with every acquisition.

When Worker A acquires the lock, it receives token 33. Worker A pauses. Worker B acquires the lock and receives token 34. Worker B writes to storage, passing token = 34. Storage updates its record and saves the latest token.

When Worker A unfreezes and attempts to write with token 33, the storage layer checks the incoming token against the highest token it has processed. Because 33 < 34, storage rejects the write.

Enforcing the Fence in Storage

The lock manager merely assigns the token. Safety is enforced at the database or object store.

In a relational database like PostgreSQL, enforcing this is a single conditional clause:

UPDATE settlement_batches
SET status = 'COMPLETED',
    summary_data = $1,
    fence_token = $2
WHERE id = $3
  AND fence_token < $2;

If Worker A wakes up late, the query updates zero rows. The application catches the empty update result, realizes it lost its lease, and rolls back cleanly without corrupting state.

If you are using consensus-backed coordination like ZooKeeper recipes, sequential ephemeral znodes produce monotonic counter suffixes automatically. With systems like etcd, the raft_index or revision number serves as a built-in fencing token.

The Irony: Do You Even Need the Lock?

Look closely at the SQL query above:

WHERE id = $3 AND fence_token < $2;

This is optimistic concurrency control. Once your database enforces conditional writes based on version numbers, you must ask a serious question: why are you paying for a separate distributed lock in the first place?

If your downstream storage can evaluate preconditions, you can often ditch the distributed lock coordinator altogether:

• **Use Optimistic Concurrency Control directly:** Store an integer version on the row. Worker fetches row with version = 5, processes data, and executes UPDATE ... WHERE version = 5. If another worker updated it first, the statement fails harmlessly.

• **Use Database Advisory Locks:** If you already run PostgreSQL, pg_try_advisory_xact_lock(batch_id) ties the lock directly to the database connection and transaction boundary. If the worker crashes or the connection drops, PostgreSQL releases the lock immediately.

• **Distinguish Efficiency from Correctness:** As Kleppmann pointed out, locks serve two purposes: efficiency and correctness. If two workers computing the same task merely wastes a few CPU cycles, a simple Redis lock is fine. If concurrent execution corrupts financial records or writes dirty files, timed leases alone are unsafe.

When correctness matters, never rely on a timed lease without a fencing check at the boundary where bytes hit the disk.