Field Note 48 current

Emit async work into the WAL, not a job table

By: Theo Zourzouvillys
Published: June 29, 2026
Tags: architectureinfradatamessagingreliability

TL;DR

When the thing that should trigger async work already happens inside your database — a row is written, a state flips — don’t dual-write to a queue and don’t poll a job table. Ride the write-ahead log. Postgres’s pg_logical_emit_message(transactional => true, prefix, payload)PostgreSQL — pg_logical_emit_message()Writes a logical-decoding message straight into the WAL: a routing prefix plus an opaque text/bytea payload, returning the LSN. With transactional => true it is decoded only if the surrounding transaction commits; with false it is emitted immediately regardless of outcome. No table, no heap tuple — the message lives only in the WAL stream.postgresql.org ↗ writes an event straight into the WAL inside your transaction — no table, no row to insert, no dead tuple to vacuum — and a logical-replication consumer (a “WAL listener”) reads the decoded stream, processes it statefully, and pushes the work onward (into a queue, a search index, another datastore).

You get the transactional outbox guarantee — the event is emitted atomically with the write, or not at all — without the outbox table. And because the expensive processing lives in the consumer, the load stays off your primary: no polling loop, no job-table churn, no vacuum tail.

The catch you must respect: a replication slot pins WAL until the consumer confirms it, so a dead or lagging listener can fill the primary’s disk and take it down. Bound it (max_slot_wal_keep_sizePostgreSQL — max_slot_wal_keep_sizeCaps how much WAL a replication slot may pin. Past the limit Postgres invalidates the lagging slot so it can recycle WAL — sacrificing the slot to protect the primary's disk, rather than letting an unconfirmed slot fill it. The essential safety valve for any logical-decoding consumer.postgresql.org ↗), alert on slot lag, and make the consumer idempotent — delivery is at-least-once.

Context

This is the concrete mechanism behind ZFN-24: one transactional store per write, changes propagated asynchronously. The whole difficulty of “propagate asynchronously” is the seam between the commit and the propagation. Two common ways to cross it are both bad:

Dual write. Commit the row, then publish to a queue as a second operation. The two aren’t atomic — the commit lands and the publish fails (the process dies, the broker is down), and your systems silently diverge. This is the canonical distributed-systems footgun, and no amount of retry-after-commit fully closes it.
Poll a table. Write a jobs row, run a worker that loops SELECT … FOR UPDATE SKIP LOCKED. Correct, but it dumps the entire async workload — the inserts, the dead rows, the constant polling, the vacuum — onto your primary, the one database you least want to load.

The transactional outboxTransactional Outbox patternWrite an 'outbox' record in the same database transaction as the business change so the two commit atomically, then a separate relay publishes the record. It solves the dual-write problem (you must not commit a row and publish to a queue as two non-atomic steps). CDC off the WAL is the clean, poll-free relay.microservices.io ↗ fixes the atomicity: write an outbox row in the same transaction as the business change so they commit together, then a separate relay publishes it. Reading that outbox via change-data-capture off the WAL — rather than polling it — is the clean version.

pg_logical_emit_messagePostgreSQL — pg_logical_emit_message()Writes a logical-decoding message straight into the WAL: a routing prefix plus an opaque text/bytea payload, returning the LSN. With transactional => true it is decoded only if the surrounding transaction commits; with false it is emitted immediately regardless of outcome. No table, no heap tuple — the message lives only in the WAL stream.postgresql.org ↗ removes the last piece of overhead: the table itself. Every durable write in Postgres already flows through the WAL — that’s how durability and replication work. The function writes a message directly into that log: a prefix for routing and an opaque content payload, surfaced by logical decodingPostgreSQL — Logical DecodingHow Postgres streams a logically-decoded view of the WAL to consumers through a replication slot. The slot is a durable cursor: it tracks the consumer's confirmed LSN so a crashed listener resumes exactly where it left off — and, crucially, the server retains WAL until the slot confirms it, which is both the durability guarantee and the operational hazard.postgresql.org ↗ and delivered to your consumer in commit order. With transactional => true the message is decoded only if the surrounding transaction commits — so you keep the outbox’s atomicity — but it leaves no heap tuple, no index entry, nothing to vacuum. It’s the outbox without the box.

Recommendation

When the source of truth and the trigger both live in your database, emit the work into the WAL and process it off-box.

Emit transactionally. In the same transaction as the business write, call pg_logical_emit_message(true, 'order.placed', payload). It commits atomically with your change or not at all — outbox semantics, zero table. Reach for transactional => false only when you deliberately want a fire-immediately signal that ignores commit/rollback, and accept that you’ve given up the atomicity that was the whole point.
Make the payload an explicit, versioned event — not a row diff. This is a real advantage over table-based CDC: capturing row changes couples the consumer to your physical schema, so renaming a column breaks it downstream. An emitted message is an intentional event with its own contract and a version field (ZFN-14). You publish what you mean, not whatever shape your tables happen to have this quarter.
Consume through a durable logical replication slot. The slotPostgreSQL — Logical DecodingHow Postgres streams a logically-decoded view of the WAL to consumers through a replication slot. The slot is a durable cursor: it tracks the consumer's confirmed LSN so a crashed listener resumes exactly where it left off — and, crucially, the server retains WAL until the slot confirms it, which is both the durability guarantee and the operational hazard.postgresql.org ↗ is the consumer’s cursor: Postgres tracks its confirmed LSN, so the listener can crash and resume exactly where it left off — that’s the “stateful” part, and you get it for free. Decode with whatever fits (pgoutput, wal2json, Debezium), do the work, then advance the slot.
Treat the listener as a bridge, then fan out. A slot is a single ordered stream with one consumer — you do not get a queue’s competing-consumers parallelism for free. So the listener’s job is usually to validate, dedupe, and push the work into the right downstream tool: a queue for work distribution, a topic for fan-out, a journal for history, an OLAP store or search index for derived state. Choose that downstream deliberately — they are not interchangeable (ZFN-12).
Process idempotently. Delivery is at-least-once: a consumer that does its work and then dies before confirming the LSN will re-see the message on restart. Key every effect on the message’s LSN or your own idempotency key (ZFN-19) so a redelivery is a no-op rather than a double-charge.
Keep the heavy work off the primary. Run the consumer as its own process and do the expensive transformation and fan-out there, not in database queries against the primary. This is data-plane / control-plane hygiene (ZFN-16) applied to async work: the primary’s extra job is writing a WAL message it was effectively writing anyway.

From the field

The first time I reached for this, the goal was a steady firehose of state changes into a search index and an analytics store without the primary feeling a thing. The polling worker it replaced had quietly become the heaviest thing on the box — thousands of SKIP LOCKED probes an hour, most returning nothing, plus the vacuum tail of the job table. Emitting WAL messages and running one listener that fanned out into a queue dropped that load to near-zero on the primary and moved it onto a consumer I could scale and restart on its own. The part that surprised me was operational, not architectural: the day the consumer silently wedged, the slot kept pinning WAL and the primary’s disk began marching toward full. That’s the trade you’re signing up for — and exactly why max_slot_wal_keep_size and slot-lag alerting are not optional extras.

Consequences

Easier:

Atomic event emission with no outbox table — no inserts, no deletes, no dead rows, no vacuum on the work-trigger path.
The async load moves off the primary: no polling, no job-table churn; the expensive processing runs in a consumer you scale and restart independently.
The consumer is resumable by construction — the slot’s confirmed LSN is durable state, so a crash resumes precisely where it stopped, with no checkpoint table of your own.
You emit an explicit event contract, decoupled from physical schema, instead of leaking row shapes to every downstream system.

Harder:

A replication slot pins WAL until it’s confirmed — a dead or lagging consumer grows WAL without bound and can fill the primary’s disk and take it down. You must cap it (max_slot_wal_keep_sizePostgreSQL — max_slot_wal_keep_sizeCaps how much WAL a replication slot may pin. Past the limit Postgres invalidates the lagging slot so it can recycle WAL — sacrificing the slot to protect the primary's disk, rather than letting an unconfirmed slot fill it. The essential safety valve for any logical-decoding consumer.postgresql.org ↗, which invalidates the slot rather than filling the disk) and alert on slot lag. This is a sharper footgun than a bloated job table: the failure mode is the primary, not the worker.
A slow consumer is a backpressure problem you now own end-to-end — there’s no broker absorbing the burst, so flow control and bounded retention matter (ZFN-13).
At-least-once, single ordered stream. No exactly-once and no free parallelism — you need idempotent consumers and you fan out downstream for throughput.
Messages are ephemeral. Once the slot advances past a WAL message and the WAL recycles, it’s gone: no SELECT over history, no replay from the source. If you need durable history or audit, write a journal too (ZFN-12) — the WAL message is a trigger, not a system of record.
More operating surface. wal_level = logical (a modest WAL increase), a decoding plugin, slot monitoring, and failover handling — on older Postgres or some managed platforms a failover can drop a slot’s position and force a reconcile (PG 16+ failover slots help). Decoding itself also costs the primary some walsender CPU, so the win is avoiding the polling, the table, and the processing — not that the primary does literally nothing.

New obligations:

Monitor every slot’s lag and bound its WAL retention from day one. Treat an unconfirmed, growing slot as a page-worthy condition, not a dashboard curiosity.
Own the event payload as a versioned contract, and define the dead-letter / poison-message path for messages the consumer genuinely cannot process — otherwise one bad message stalls the whole ordered stream behind it.

References

pg_logical_emit_message() — emit a message into the WAL, transactionally or not; the table-free outbox primitive.
PostgreSQL Logical Decoding — replication slots as durable cursors, and the WAL-retention behaviour you must operate.
Transactional Outbox pattern — why you emit atomically with the write instead of dual-writing to a queue.
max_slot_wal_keep_size — the safety valve that protects the primary from a lagging slot.
ZFN-24 — the principle this implements: one transactional store, propagate changes asynchronously.
ZFN-12 — the WAL stream is journal-shaped; choose the downstream queue / topic / journal deliberately.
ZFN-19 — at-least-once delivery demands idempotent processing.
ZFN-16 and ZFN-13 — keep processing off the primary, and handle the backpressure the slot exposes.
ZFN-14 — the emitted message is an event contract; version it rather than leaking your schema.

Changelog

2026-06-29: First published as a Field Note.