Journal the write, apply it in micro-batches

Context

A large share of the writes in a typical service are not things anyone is waiting on: last_seen_at touches, usage counters, telemetry and audit events, search-index updates, denormalised read models, syncs into a third-party system. They sit in the request path because that is the default shape of code, not because anything requires it. You pay for that three times over: latency on every request, the availability of your endpoint welded to the availability of the store, and a traffic burst converted directly into a write burst.

The store dislikes it as much as you should. Row-at-a-time is a horrible way to write to anything, and the reason is arithmetic rather than taste: per-statement round trip, per-transaction commit, per-commit durable flush, and the overhead per transaction dominates the cost of the row itselfPostgreSQL — Populating a DatabaseThe manual's own guidance on bulk writes: batch many inserts into a single transaction (or COPY) rather than committing each one, because per-transaction commit overhead (round trip, WAL flush, fsync) dominates the actual cost of the row. The mechanical reason micro-batching wins.postgresql.org ↗. Put numbers on it. A hundred inserts at a millisecond of commit apiece is a tenth of a second of pure overhead carrying maybe a few kilobytes of actual data. The same hundred rows as one statement is one round trip and one flush. The same arithmetic holds for an external API, where each call is a TLS round trip and a slice of somebody’s rate limit.

This is the mirror image of ZFN-48Field Note · currentZFN-48 — Emit async work into the WAL, not a job tableWhen a DB write should trigger async work, ride the WAL instead of dual-writing or polling a job table. pg_logical_emit_message emits the event transactionally — outbox semantics, no table. A WAL listener consumes it statefully and fans out, keeping load off the primary.Why it's cited here: The mirror image of this note: there the database is the source of async work and the WAL carries it outward; here the database is the sink and the journal sits in front of it.Open ZFN-48 →. There the write has already happened and the database is the source of async work, emitted outward through the WAL. here the write has not happened yet and does not need to, and the database is the sink, so the journal sits in front of it, absorbing writes and feeding them in at a rate and shape the store actually likes. Same primitive, pointed the other way.

The reason this is not simply “put it on a queue” is ZFN-12Field Note · currentZFN-12 — Queues, topics, and journals are different tools — don't conflate themQueues (competing consumers), topics (fan-out), and journals (ordered, replayable logs) give different guarantees. Don't conflate them; a pipeline often uses several. Prefer journals over topics, but not where head-of-line blocking hurts. With queues, bound the concurrency.Why it's cited here: The buffer is specifically a journal and not a queue. The ordering and replay it gives you are the whole reason a batched apply can be made safe.Open ZFN-12 →. A queue gives you independent per-item progress and no ordering, which is exactly wrong when the items are mutations of the same rows. What you want is a journal: ordered, durable, with a consumer that tracks its own offset and can be replayed after a bad deploy (DDIA ch. 11Martin Kleppmann — Designing Data-Intensive Applications, ch. 11The chapter on stream processing: logs as ordered, replayable records; consumer offsets as durable cursors; and why ordering guarantees are per-partition rather than global. The reference text for why a buffered write pipeline behaves the way it does.dataintensive.net ↗).

Recommendation

Buffer writes that nobody is waiting on into an ordered journal, and apply them in micro-batches. Treat ordering as a design obligation and not as a property you hope survives.

Decide per write, with one test. Does anything the caller does next depend on this having already happened? If yes, it is synchronous. If no, buffer it. That test, and not “is it important?”, is the one that partitions correctly. An audit event can be extremely important and still not need to be written before the response goes out.

Make the buffer as durable as the data deserves, and say which you chose. An in-process ring buffer is a decision to lose the last few seconds of writes on a crash. That is a perfectly good trade for view counters and a terrible one for audit records. Name the loss budget out loud, because the failure mode of not naming it is discovering it during an incident. Whatever you choose, the buffer has to be flushed on shutdown, which makes it part of the drain sequence in ZFN-60Field Note · currentZFN-60 — Drain before you die: graceful shutdown is a protocolSIGTERM isn't an emergency — it's every deploy and scale-in. Shutdown is a protocol: stop attracting work, drain while the balancer catches up, hand back in-flight work, release leases, exit before SIGKILL. But graceful is only the optimisation — crash-safe is the requirement.Open ZFN-60 → rather than something the process can exit without.

Apply in micro-batches, and commit the offset last. Take up to N records or wait up to T milliseconds, whichever comes first (the timer stops a quiet period stranding records, the count stops a busy one building an unbounded transaction). Apply the batch in a single transaction, then advance the offset after that transaction commits. That ordering is what makes the pipeline at-least-once rather than at-most-once, and at-least-once means the apply has to be idempotent (ZFN-19Field Note · currentZFN-19 — Annotate read-only and idempotent endpoints; make every mutation idempotentAnnotate every endpoint as read-only (safe) or idempotent, in the schema, so infrastructure can retry, route to replicas, and cache safely. Make every state-changing endpoint idempotent (idempotency keys for create/charge/send); a non-idempotent retry double-applies.Open ZFN-19 →), because a crash between commit and offset advance will re-apply the batch.

Take order from the journal, never from the clock. This is the part that quietly goes wrong. Once records from many producers are sitting in a buffer, something has to decide what order they apply in, and the tempting field is the created_at each producer stamped on the way in. That is not an ordering. It is several machines’ disagreeing opinions of now, and the skew is small enough that it always works in testing (ZFN-59Field Note · currentZFN-59 — Two clocks: monotonic for durations, wall time for recordsYou have two clocks. Wall time names moments — and it jumps, slews, and runs backwards. The monotonic clock measures elapsed time — and means nothing across machines or reboots. Every timeout, lease, and cross-machine ordering bug is one clock doing the other's job.Why it's cited here: The rule that makes buffered writes orderable at all: sequence comes from one writer's log position, never from each producer's opinion of what time it is.Open ZFN-59 →). The sequence that means something is the journal’s own: the offset assigned by the single writer that accepted the append (ZFN-24Field Note · currentZFN-24 — One transactional store per write; propagate changes asynchronouslyCommit each logical write to exactly one transactional store; update other systems via reliable ordered async events — never a synchronous write across two stores, and never 2PC. With a relational primary the WAL is your replayable journal; write events into the same transaction.Why it's cited here: Buffering is how 'propagate changes asynchronously' actually gets implemented on the write path, without giving up the single transactional store.Open ZFN-24 →). Coalescing inside a batch (collapsing five updates to one row into a last-write-wins) has to resolve by that offset too, or you have built a machine that reliably applies the oldest update last.

Partition by entity key. Parallelise across keys, never within one. One consumer per journal is usually not enough throughput, and the safe way to scale is to shard by the key whose order you care about: user id, tenant id, document id. Round-robin sharding of a batch across workers is the classic version of this bug. Two mutations of the same row land in different workers and race, and the winner is whichever one the scheduler felt like. Sharding by key keeps operations on a given entity in a single ordered lane, and lets unrelated entities get on with it in parallel.

Never run two paths to the same entity. This is where DELETE overtakes INSERT. If the create is buffered and the delete is synchronous (because deletes felt urgent, or because they were written by someone else, or because one call arrives through the API and the other through an admin tool) then the delete finds nothing to delete, the buffered insert lands afterwards, and the row is back from the dead with nobody watching. The fix is not more care, it is structural: route every mutation of an entity through the same lane, or make the apply conditional on a version the producer supplied so a stale operation is discarded rather than applied (ZFN-25Field Note · currentZFN-25 — Track the version a client has seen for read-your-writesFor read-your-writes across backends, track the latest version a client has seen — a token or vector clock. Return it on write; reads then go to a backend at least that fresh. Hold it client-side (a token they present) or server-side (a gateway tracks the session and routes).Open ZFN-25 →). Two paths and hope is not a design.

Bound the buffer, and decide what “full” means. A buffer that grows without limit relocates the outage rather than preventing it. Cap it, monitor consumer lag as a first-class signal, and choose deliberately whether a full buffer sheds the write or pushes back on the producer (ZFN-13Field Note · currentZFN-13 — Fail fast and push back: retries, load shedding, and flow controlBuild client retries (backoff, jitter, Retry-After) from day one. Under overload, shed fast and push the failure back to the source to retry — don't retry internally and amplify it. Flow-control everywhere, bound every queue, and don't take more work than you can finish in time.Open ZFN-13 →). For counters, dropping is often correct. For audit it never is, which usually means audit gets a durable journal rather than an in-memory one.

Consequences

Easier:

• Request latency drops by whatever the write cost, and endpoints stop failing because a downstream store or third-party API is having a bad minute.
• The store sees far fewer, far larger transactions, which is the shape it is efficient at, instead of a storm of single-row commits.
• Bursts turn into backlog. The journal absorbs the spike and the consumer drains at a rate you choose, rather than passing the spike straight through.
• Coalescing becomes possible at all. A thousand touches of one counter inside a window can become one write, which is simply not available when each is applied inline.
• A bad consumer deploy is recoverable by replay, because a journal keeps history and a queue does not.

Harder:

• You have moved the error contract. The caller got 200 before the write succeeded, so a failure at apply time has nobody to tell. It needs somewhere to go (a dead-letter path and an alert), and the API documentation needs to stop implying the write already happened (ZFN-58Field Note · currentZFN-58 — Errors are part of the contractError paths are the half of your API clients depend on most, and usually the half nobody designed. Enumerate error codes in the schema like any other type: stable code, retryable-or-not, whose fault, structured params. Machines branch on codes — anyone parsing prose is broken.Open ZFN-58 →).
• Read-your-writes is gone for anything buffered. A caller that writes and immediately reads sees the old value, and that surprise is usually worse than the latency you saved (ZFN-25Field Note · currentZFN-25 — Track the version a client has seen for read-your-writesFor read-your-writes across backends, track the latest version a client has seen — a token or vector clock. Return it on write; reads then go to a backend at least that fresh. Hold it client-side (a token they present) or server-side (a gateway tracks the session and routes).Open ZFN-25 →).
• Constraints no longer reject at request time. A uniqueness violation surfaces in the consumer, minutes later, on a request that has already been answered successfully.
• Ordering is now something you own explicitly, and its failures are the quiet kind: a resurrected row, a counter that drifts, a stale value that wins. none of them raise an error.
• One more moving part to run, with its own lag, its own backlog, and its own way of being down.

New obligations:

• Monitor consumer lag and buffer depth, and page on them. An unnoticed stalled consumer is a silent, growing pile of writes that have been acknowledged and not performed.
• Define the dead-letter path before launch, and make sure a poison record cannot wedge the lane behind it forever.
• Write down the durability choice, meaning what a crash loses, where the next person will find it.

If that list of obligations reads as long, that is the honest price and it is worth checking you want to pay it. Buffering is not free, it is a trade: you are buying latency and load-levelling with ordering work and a worse error contract. For a last_seen_at touch that is an obvious yes. For anything a human will be shown as confirmation, think a lot harder!

References

• Designing Data-Intensive Applications, ch. 11 on logs, consumer offsets, and why ordering guarantees are per-partition rather than global.
• PostgreSQL: Populating a Database, the mechanical case for batching, where per-transaction overhead dominates per-row cost.
• ZFN-48Field Note · currentZFN-48 — Emit async work into the WAL, not a job tableWhen a DB write should trigger async work, ride the WAL instead of dual-writing or polling a job table. pg_logical_emit_message emits the event transactionally — outbox semantics, no table. A WAL listener consumes it statefully and fans out, keeping load off the primary.Why it's cited here: The mirror image of this note: there the database is the source of async work and the WAL carries it outward; here the database is the sink and the journal sits in front of it.Open ZFN-48 →, the same journal pointed the other way: the database as the source of async work rather than its sink.
• ZFN-12Field Note · currentZFN-12 — Queues, topics, and journals are different tools — don't conflate themQueues (competing consumers), topics (fan-out), and journals (ordered, replayable logs) give different guarantees. Don't conflate them; a pipeline often uses several. Prefer journals over topics, but not where head-of-line blocking hurts. With queues, bound the concurrency.Why it's cited here: The buffer is specifically a journal and not a queue. The ordering and replay it gives you are the whole reason a batched apply can be made safe.Open ZFN-12 → on why the buffer is a journal and not a queue.
• ZFN-59Field Note · currentZFN-59 — Two clocks: monotonic for durations, wall time for recordsYou have two clocks. Wall time names moments — and it jumps, slews, and runs backwards. The monotonic clock measures elapsed time — and means nothing across machines or reboots. Every timeout, lease, and cross-machine ordering bug is one clock doing the other's job.Why it's cited here: The rule that makes buffered writes orderable at all: sequence comes from one writer's log position, never from each producer's opinion of what time it is.Open ZFN-59 → on never ordering cross-machine events by wall-clock timestamps.
• ZFN-24Field Note · currentZFN-24 — One transactional store per write; propagate changes asynchronouslyCommit each logical write to exactly one transactional store; update other systems via reliable ordered async events — never a synchronous write across two stores, and never 2PC. With a relational primary the WAL is your replayable journal; write events into the same transaction.Why it's cited here: Buffering is how 'propagate changes asynchronously' actually gets implemented on the write path, without giving up the single transactional store.Open ZFN-24 →, the single writer whose sequence is the only real ordering you have.
• ZFN-19Field Note · currentZFN-19 — Annotate read-only and idempotent endpoints; make every mutation idempotentAnnotate every endpoint as read-only (safe) or idempotent, in the schema, so infrastructure can retry, route to replicas, and cache safely. Make every state-changing endpoint idempotent (idempotency keys for create/charge/send); a non-idempotent retry double-applies.Open ZFN-19 → on at-least-once delivery demanding an idempotent apply.
• ZFN-13Field Note · currentZFN-13 — Fail fast and push back: retries, load shedding, and flow controlBuild client retries (backoff, jitter, Retry-After) from day one. Under overload, shed fast and push the failure back to the source to retry — don't retry internally and amplify it. Flow-control everywhere, bound every queue, and don't take more work than you can finish in time.Open ZFN-13 → on a bounded buffer, and what happens when it fills.
• ZFN-60Field Note · currentZFN-60 — Drain before you die: graceful shutdown is a protocolSIGTERM isn't an emergency — it's every deploy and scale-in. Shutdown is a protocol: stop attracting work, drain while the balancer catches up, hand back in-flight work, release leases, exit before SIGKILL. But graceful is only the optimisation — crash-safe is the requirement.Open ZFN-60 →, where the buffer is part of the drain.
• ZFN-58Field Note · currentZFN-58 — Errors are part of the contractError paths are the half of your API clients depend on most, and usually the half nobody designed. Enumerate error codes in the schema like any other type: stable code, retryable-or-not, whose fault, structured params. Machines branch on codes — anyone parsing prose is broken.Open ZFN-58 → and ZFN-25Field Note · currentZFN-25 — Track the version a client has seen for read-your-writesFor read-your-writes across backends, track the latest version a client has seen — a token or vector clock. Return it on write; reads then go to a backend at least that fresh. Hold it client-side (a token they present) or server-side (a gateway tracks the session and routes).Open ZFN-25 →, the two contracts buffering quietly changes.

Changelog

• 2026-08-19: First published as a Field Note.