---
id: 59
title: "Two clocks: monotonic for durations, wall time for records"
kind: note
status: current
date: 2026-08-12
authors:
  - "Theo Zourzouvillys"
tags: [correctness, reliability, architecture, consistency]
references:
  - id: nonow
    title: "There Is No Now (Justin Sheehy, ACM Queue, 2015)"
    url: https://queue.acm.org/detail.cfm?id=2745385
    abstract: "The clearest short argument that 'now' is not a coherent concept across a distributed system: light-speed delays, clock error, and failure make simultaneity unknowable, so correct systems must be designed around uncertainty in time rather than pretending a shared present exists."
  - id: cfleap
    title: "How and why the leap second affected Cloudflare DNS (2017)"
    url: https://blog.cloudflare.com/how-and-why-the-leap-second-affected-cloudflare-dns/
    abstract: "A production postmortem of wall-clock time going backwards: a leap second made time.Now() return a smaller value than before, a subtraction produced a negative duration, and DNS resolution failed — the canonical worked example of why elapsed time must never come from the wall clock."
  - id: clockgettime
    title: "clock_gettime(2) — Linux manual page"
    url: https://man7.org/linux/man-pages/man2/clock_gettime.2.html
    abstract: "The system-call level statement of the distinction: CLOCK_REALTIME is wall time and can jump forwards or backwards on adjustment; CLOCK_MONOTONIC never goes backwards but has an arbitrary epoch, making it meaningful only for measuring intervals within one running system."
summary: "You have two clocks. Wall time names moments — and it jumps, slews, and runs backwards. The monotonic clock measures elapsed time — and means nothing across machines or reboots. Every timeout, lease, and cross-machine ordering bug is one clock doing the other's job."
supersedes: null
superseded_by: null
aliases: []
crossrefs:
  ZFN-37: "Leases are where the two-clock rule earns its keep: expiry decided by the single authority's clock and fenced with tokens, because holders' clocks cannot agree."
  ZFN-24: "The WAL is the answer to 'what order did things happen in?' — a sequence assigned by one writer, which is exactly what wall-clock timestamps across machines can never be."
  ZFN-25: "Version tokens exist because 'newer timestamp wins' is not a consistency model; freshness is tracked by explicit versions, not by comparing clocks."
---

## TL;DR

**Your machine has two clocks, and almost every time-related production bug is one of them doing
the other one's job.**

The **wall clock** (`CLOCK_REALTIME`, `Date.now()`, `time.Now()`) tells you *what time it is*:
it's for timestamping records, showing humans, and honouring calendars. It is also adjusted
behind your back — NTP steps and slews, leap seconds, VM pauses — and it can and does
[run backwards](ref:cfleap).

The **monotonic clock** (`CLOCK_MONOTONIC`) tells you *how much time has passed*: it never goes
backwards, and it's the only legitimate source for durations — timeouts, latency measurements,
rate limiters, retry backoff, in-process lease checks. Its reading is meaningless outside the
running process that took it ([clock_gettime(2)](ref:clockgettime)).

Three rules cover nearly everything:

- **Durations from the monotonic clock; instants from the wall clock, in UTC.**
- **Never order cross-machine events by wall-clock timestamps.** Order comes from a single
  writer's sequence ([ZFN-24](/zfn/24-one-transactional-store-per-write/)), a version token
  ([ZFN-25](/zfn/25-read-your-writes-version-token/)), or explicit causality — never from
  comparing two machines' opinions of "now" ([there is no now](ref:nonow)).
- **Never let correctness depend on two machines' clocks agreeing.** Clock error is usually
  small — which is the trap: designs that need it to be small *always* work in testing.

## Context

Time looks like a solved problem because every language hands you a `now()` and it's almost
always almost right. The failure modes hide in the "almost":

- **Wall time goes backwards.** An NTP step after drift, a [leap second](ref:cfleap), a VM
  migration — and suddenly `end - start` is negative. Cloudflare's DNS fell over exactly this
  way: a negative duration from two wall-clock reads, in code that could never have imagined
  it. Any timeout, cache TTL, or profiler built on wall-clock subtraction carries this bug,
  latent, until the clock steps under load.
- **Clocks disagree across machines — by more than you think, exactly when you least want.**
  Well-run fleets sit within milliseconds; the tail — an unsynced host, a paused VM, a failed
  NTP daemon — sits at seconds or minutes. So "the event with the later timestamp happened
  later" is a heuristic that's true until it's an incident: last-write-wins quietly discarding
  the *earlier*-stamped later write; a "newer" cache entry losing to a stale one stamped by a
  fast clock; audit logs that swear the response left before the request arrived.
- **Certificates, tokens, and schedules live on other people's clocks.** An `iat` or `nbf`
  minted by a fast clock is "in the future" for everyone else and gets rejected
  ([ZFN-6](/zfn/6-sender-constrained-tokens-dpop/)-style proofs and short-lived credentials
  ([ZFN-9](/zfn/9-no-long-lived-cloud-keys/)) both live and die by this); the fix — validation
  leeway — is a designed tolerance, not a hack, and it has to be chosen deliberately.
- **Civil time is a policy, not a physics.** "Every day at 09:00 in Amsterdam" is not an
  instant: timezone rules change by legislation, DST skips and repeats an hour. Store *future
  civil intent* as local-time-plus-zone and resolve it late; store *past instants* as UTC.
  Converting a future appointment to UTC at write time silently bakes in today's timezone law.

> [!aside]
>
> The interview-question version: a request arrives with a wall-clock timestamp claiming it was
> sent 400ms ago. What do you know? Nothing. Sender's clock fast, slow, stepped in between —
> indistinguishable. Cross-machine wall time answers "roughly when, for the humans" and nothing
> else. Everything sharper needs a sequence, a token, or a single authority's clock.

## Recommendation

**Give each clock its one job, and design the places they meet.**

- **Every duration from the monotonic clock.** Timeouts, deadlines-as-remaining-budget
  ([ZFN-61](/zfn/61-propagate-the-deadline/)), backoff ([ZFN-13](/zfn/13-load-shedding-and-flow-control/)),
  latency metrics, token buckets. Audit the codebase for `now() - then` on wall time; each one
  is the Cloudflare bug waiting for a leap second or an NTP step.

- **Every recorded instant in UTC**, RFC 3339 on the wire, converted to local time only at the
  human's screen. (This site's own build formats note dates with an explicitly-UTC formatter
  because a bare `YYYY-MM-DD` parsed in a western timezone renders as the previous day — the
  small, dumb version of every wall-clock bug: an instant re-interpreted by a second clock.)

- **Order by sequence, not by stamp.** Within one store, the transaction log already orders
  everything ([ZFN-24](/zfn/24-one-transactional-store-per-write/),
  [ZFN-48](/zfn/48-emit-async-work-into-the-wal/)); across stores, carry version tokens
  ([ZFN-25](/zfn/25-read-your-writes-version-token/)) or event sequence numbers
  ([ZFN-12](/zfn/12-queues-topics-journals/)). Wall-clock timestamps on events are *metadata
  for humans* — keep writing them, stop *deciding* with them.

- **Expiry is decided by the single authority that granted it.** A lease's TTL is checked
  against the lock service's clock, nobody else's, and side effects are fenced so a holder
  whose clock says "still mine" can't corrupt anything
  ([ZFN-37](/zfn/37-every-lock-is-a-lease/)). The same shape everywhere: one clock owns each
  deadline; everyone else asks, or holds a fencing token that makes their opinion irrelevant.

- **Build in leeway wherever another party's clock is validated.** Token `nbf`/`iat`/`exp`,
  signature windows, replay caches: a small, documented tolerance (tens of seconds), chosen
  against your threat model rather than inherited from a library default. Zero leeway is an
  outage on someone's fast clock; huge leeway is a replay window
  ([ZFN-7](/zfn/7-sign-the-message/)).

- **Run NTP everywhere and alert on drift** — not to make any of the above safe (it can't),
  but because well-synced clocks make the *metadata* trustworthy enough for debugging, and
  drift alerts catch the pathological host before it stars in an incident.

## Consequences

**Easier:**

- **A whole genus of heisenbug dies**: negative durations, timeouts that fire instantly or
  never, metrics spikes at 2 a.m. on patch Tuesday, the once-a-year leap-second scramble.
- **Ordering arguments become checkable.** "It's ordered because the journal says so" replaces
  "the timestamps looked right," and code review can enforce the difference.
- **Incident timelines assembled across machines stop lying to you** — because nothing
  correctness-shaped depended on them, drift is a nuisance, not a corruption.

**Harder:**

- **You lose the easy answer.** "Just compare timestamps" was free; sequences, versions, and
  fencing tokens are machinery you now have to carry
  ([ZFN-20](/zfn/20-deliberate-complexity-is-often-simpler/) — the deliberate design with fewer
  surprises).
- **Two representations of time thread through the code** — typed as such if the language
  allows — and the discipline needs review-level enforcement, because every new hire arrives
  fluent in exactly the mistake this note bans.
- **Future civil time stays genuinely annoying**: local-time-plus-zone storage, late
  resolution, and re-resolution when tzdata changes. There's no clean escape; the note's
  contribution is only that you stop *also* using it for machine scheduling.

## References

- [ZFN-37](/zfn/37-every-lock-is-a-lease/) — expiry owned by one authority, fenced against
  everyone else's clock; this note's rules made concrete in the hardest case.
- [ZFN-24](/zfn/24-one-transactional-store-per-write/) and
  [ZFN-25](/zfn/25-read-your-writes-version-token/) — where order and freshness actually come
  from, once timestamps are demoted.
- [ZFN-61](/zfn/61-propagate-the-deadline/) — deadlines carried as remaining budget precisely so
  no two machines' wall clocks need to agree.
- [There Is No Now](ref:nonow) — the conceptual ground; [Cloudflare's leap-second
  postmortem](ref:cfleap) — the empirical one; [clock_gettime(2)](ref:clockgettime) — the two
  clocks, stated by the kernel.

## Changelog

- **2026-08-12**: First published as a Field Note.