Two clocks: monotonic for durations, wall time for records

Context

Time looks like a solved problem because every language hands you a now() and it’s almost always almost right. The failure modes hide in the “almost”:

• Wall time goes backwards. An NTP step after drift, a leap secondHow and why the leap second affected Cloudflare DNS (2017)A production postmortem of wall-clock time going backwards: a leap second made time.Now() return a smaller value than before, a subtraction produced a negative duration, and DNS resolution failed — the canonical worked example of why elapsed time must never come from the wall clock.blog.cloudflare.com ↗, a VM migration — and suddenly end - start is negative. Cloudflare’s DNS fell over exactly this way: a negative duration from two wall-clock reads, in code that could never have imagined it. Any timeout, cache TTL, or profiler built on wall-clock subtraction carries this bug, latent, until the clock steps under load.
• Clocks disagree across machines — by more than you think, exactly when you least want. Well-run fleets sit within milliseconds; the tail — an unsynced host, a paused VM, a failed NTP daemon — sits at seconds or minutes. So “the event with the later timestamp happened later” is a heuristic that’s true until it’s an incident: last-write-wins quietly discarding the earlier-stamped later write; a “newer” cache entry losing to a stale one stamped by a fast clock; audit logs that swear the response left before the request arrived.
• Certificates, tokens, and schedules live on other people’s clocks. An iat or nbf minted by a fast clock is “in the future” for everyone else and gets rejected (ZFN-6Field Note · currentZFN-6 — Bind tokens to a key: sender-constrained tokens (DPoP)A bearer token grants access to whoever holds it — steal it, replay it. Bind the token to a holder key (DPoP, RFC 9449) so using it requires proving possession of a private key the token names. A stolen token alone becomes useless.Open ZFN-6 →-style proofs and short-lived credentials (ZFN-9Field Note · currentZFN-9 — No long-lived cloud keys; workloads authenticate by federated identityNo static AWS or GCP keys anywhere — not in code, secret stores, or env. Workloads use their runtime's own identity and cross clouds by exchanging it (OIDC) for short-lived credentials via federation. Static keys are a documented carve-out only.Open ZFN-9 →) both live and die by this); the fix — validation leeway — is a designed tolerance, not a hack, and it has to be chosen deliberately.
• Civil time is a policy, not a physics. “Every day at 09:00 in Amsterdam” is not an instant: timezone rules change by legislation, DST skips and repeats an hour. Store future civil intent as local-time-plus-zone and resolve it late; store past instants as UTC. Converting a future appointment to UTC at write time silently bakes in today’s timezone law.

Recommendation

Give each clock its one job, and design the places they meet.

• Every duration from the monotonic clock. Timeouts, deadlines-as-remaining-budget (ZFN-61Field Note · currentZFN-61 — Propagate the deadlineEvery request has a deadline whether you set one or not — the caller's patience. Make it explicit at the edge, carry it as remaining budget on every hop, check it before expensive steps, and cancel downstream when it dies. Work past the deadline is the fuel of cascading collapse.Open ZFN-61 →), backoff (ZFN-13Field Note · currentZFN-13 — Fail fast and push back: retries, load shedding, and flow controlBuild client retries (backoff, jitter, Retry-After) from day one. Under overload, shed fast and push the failure back to the source to retry — don't retry internally and amplify it. Flow-control everywhere, bound every queue, and don't take more work than you can finish in time.Open ZFN-13 →), latency metrics, token buckets. Audit the codebase for now() - then on wall time; each one is the Cloudflare bug waiting for a leap second or an NTP step.

• Every recorded instant in UTC, RFC 3339 on the wire, converted to local time only at the human’s screen. (This site’s own build formats note dates with an explicitly-UTC formatter because a bare YYYY-MM-DD parsed in a western timezone renders as the previous day — the small, dumb version of every wall-clock bug: an instant re-interpreted by a second clock.)

• Order by sequence, not by stamp. Within one store, the transaction log already orders everything (ZFN-24Field Note · currentZFN-24 — One transactional store per write; propagate changes asynchronouslyCommit each logical write to exactly one transactional store; update other systems via reliable ordered async events — never a synchronous write across two stores, and never 2PC. With a relational primary the WAL is your replayable journal; write events into the same transaction.Why it's cited here: The WAL is the answer to 'what order did things happen in?' — a sequence assigned by one writer, which is exactly what wall-clock timestamps across machines can never be.Open ZFN-24 →, ZFN-48Field Note · currentZFN-48 — Emit async work into the WAL, not a job tableWhen a DB write should trigger async work, ride the WAL instead of dual-writing or polling a job table. pg_logical_emit_message emits the event transactionally — outbox semantics, no table. A WAL listener consumes it statefully and fans out, keeping load off the primary.Open ZFN-48 →); across stores, carry version tokens (ZFN-25Field Note · currentZFN-25 — Track the version a client has seen for read-your-writesFor read-your-writes across backends, track the latest version a client has seen — a token or vector clock. Return it on write; reads then go to a backend at least that fresh. Hold it client-side (a token they present) or server-side (a gateway tracks the session and routes).Why it's cited here: Version tokens exist because 'newer timestamp wins' is not a consistency model; freshness is tracked by explicit versions, not by comparing clocks.Open ZFN-25 →) or event sequence numbers (ZFN-12Field Note · currentZFN-12 — Queues, topics, and journals are different tools — don't conflate themQueues (competing consumers), topics (fan-out), and journals (ordered, replayable logs) give different guarantees. Don't conflate them; a pipeline often uses several. Prefer journals over topics, but not where head-of-line blocking hurts. With queues, bound the concurrency.Open ZFN-12 →). Wall-clock timestamps on events are metadata for humans — keep writing them, stop deciding with them.

• Expiry is decided by the single authority that granted it. A lease’s TTL is checked against the lock service’s clock, nobody else’s, and side effects are fenced so a holder whose clock says “still mine” can’t corrupt anything (ZFN-37Field Note · currentZFN-37 — Every lock is a leaseA lock that can outlive its holder is a deadlock scheduled for later. Give every lock — including informal ones like claimed_by columns — a TTL, a named owner, and a heartbeat; make expiry automatic and server-side; fence the side effects so a stale holder can't corrupt anything.Why it's cited here: Leases are where the two-clock rule earns its keep: expiry decided by the single authority's clock and fenced with tokens, because holders' clocks cannot agree.Open ZFN-37 →). The same shape everywhere: one clock owns each deadline; everyone else asks, or holds a fencing token that makes their opinion irrelevant.

• Build in leeway wherever another party’s clock is validated. Token nbf/iat/exp, signature windows, replay caches: a small, documented tolerance (tens of seconds), chosen against your threat model rather than inherited from a library default. Zero leeway is an outage on someone’s fast clock; huge leeway is a replay window (ZFN-7Field Note · currentZFN-7 — Sign the message, not just the session (HTTP Message Signatures)A bearer token proves nothing about the request it rides on. Sign the message itself (HTTP Message Signatures, RFC 9421) — request, and ideally response — so the recipient can prove who sent this exact message and not a byte changed. Shared keys first; asymmetric better.Open ZFN-7 →).

• Run NTP everywhere and alert on drift — not to make any of the above safe (it can’t), but because well-synced clocks make the metadata trustworthy enough for debugging, and drift alerts catch the pathological host before it stars in an incident.

Consequences

Easier:

• A whole genus of heisenbug dies: negative durations, timeouts that fire instantly or never, metrics spikes at 2 a.m. on patch Tuesday, the once-a-year leap-second scramble.
• Ordering arguments become checkable. “It’s ordered because the journal says so” replaces “the timestamps looked right,” and code review can enforce the difference.
• Incident timelines assembled across machines stop lying to you — because nothing correctness-shaped depended on them, drift is a nuisance, not a corruption.

Harder:

• You lose the easy answer. “Just compare timestamps” was free; sequences, versions, and fencing tokens are machinery you now have to carry (ZFN-20Field Note · currentZFN-20 — The simplest-looking system is often the most complex to live withThe system that's simplest to stand up often isn't simplest to live with — it skips the correctness edge cases, so bugs and inconsistency surface fast. A more deliberate design has more parts but fewer surprises, and is often the simpler one over time.Open ZFN-20 → — the deliberate design with fewer surprises).
• Two representations of time thread through the code — typed as such if the language allows — and the discipline needs review-level enforcement, because every new hire arrives fluent in exactly the mistake this note bans.
• Future civil time stays genuinely annoying: local-time-plus-zone storage, late resolution, and re-resolution when tzdata changes. There’s no clean escape; the note’s contribution is only that you stop also using it for machine scheduling.

References

• ZFN-37Field Note · currentZFN-37 — Every lock is a leaseA lock that can outlive its holder is a deadlock scheduled for later. Give every lock — including informal ones like claimed_by columns — a TTL, a named owner, and a heartbeat; make expiry automatic and server-side; fence the side effects so a stale holder can't corrupt anything.Why it's cited here: Leases are where the two-clock rule earns its keep: expiry decided by the single authority's clock and fenced with tokens, because holders' clocks cannot agree.Open ZFN-37 → — expiry owned by one authority, fenced against everyone else’s clock; this note’s rules made concrete in the hardest case.
• ZFN-24Field Note · currentZFN-24 — One transactional store per write; propagate changes asynchronouslyCommit each logical write to exactly one transactional store; update other systems via reliable ordered async events — never a synchronous write across two stores, and never 2PC. With a relational primary the WAL is your replayable journal; write events into the same transaction.Why it's cited here: The WAL is the answer to 'what order did things happen in?' — a sequence assigned by one writer, which is exactly what wall-clock timestamps across machines can never be.Open ZFN-24 → and ZFN-25Field Note · currentZFN-25 — Track the version a client has seen for read-your-writesFor read-your-writes across backends, track the latest version a client has seen — a token or vector clock. Return it on write; reads then go to a backend at least that fresh. Hold it client-side (a token they present) or server-side (a gateway tracks the session and routes).Why it's cited here: Version tokens exist because 'newer timestamp wins' is not a consistency model; freshness is tracked by explicit versions, not by comparing clocks.Open ZFN-25 → — where order and freshness actually come from, once timestamps are demoted.
• ZFN-61Field Note · currentZFN-61 — Propagate the deadlineEvery request has a deadline whether you set one or not — the caller's patience. Make it explicit at the edge, carry it as remaining budget on every hop, check it before expensive steps, and cancel downstream when it dies. Work past the deadline is the fuel of cascading collapse.Open ZFN-61 → — deadlines carried as remaining budget precisely so no two machines’ wall clocks need to agree.
• There Is No NowThere Is No Now (Justin Sheehy, ACM Queue, 2015)The clearest short argument that 'now' is not a coherent concept across a distributed system: light-speed delays, clock error, and failure make simultaneity unknowable, so correct systems must be designed around uncertainty in time rather than pretending a shared present exists.queue.acm.org ↗ — the conceptual ground; Cloudflare’s leap-second postmortemHow and why the leap second affected Cloudflare DNS (2017)A production postmortem of wall-clock time going backwards: a leap second made time.Now() return a smaller value than before, a subtraction produced a negative duration, and DNS resolution failed — the canonical worked example of why elapsed time must never come from the wall clock.blog.cloudflare.com ↗ — the empirical one; clock_gettime(2)clock_gettime(2) — Linux manual pageThe system-call level statement of the distinction: CLOCK_REALTIME is wall time and can jump forwards or backwards on adjustment; CLOCK_MONOTONIC never goes backwards but has an arbitrary epoch, making it meaningful only for measuring intervals within one running system.man7.org ↗ — the two clocks, stated by the kernel.

Changelog

• 2026-08-12: First published as a Field Note.