Propagate the deadline

Context

The default, in almost every HTTP shop, is per-hop timeouts chosen independently: the gateway gives 30s, the service calls the next with its client default of 60, which calls the database with 30 more. Three consequences, all bad:

• Inner hops outlive outer ones. The gateway gave up at 30s; the database is still faithfully executing the query at minute two. Multiply by every request during an incident and the database is spending its capacity almost exclusively on abandoned work — which is why the system stays down after the triggering blip has passed (the SRE book’s cascading-failure chapterAddressing Cascading Failures (Google SRE Book)Google's field guide to overload collapse, including the specific mechanisms this note leans on: deadline propagation to stop servers doing work no caller is waiting for, retry budgets to stop amplification, and load shedding to keep goodput up when demand exceeds capacity.sre.google ↗ is substantially a catalogue of this one mistake).
• Nobody can answer “how long may this take?” The end-to-end budget is an emergent property of scattered config values, discovered empirically, during incidents.
• Timeouts stack with retries multiplicatively. Three tiers each retrying three times with independent timeouts is up to 27 attempts against the struggling dependency — retry amplification wired in at design time.

The reframe: a timeout is a local, defensive guess; a deadline is a fact about the request — after instant T, the answer has no value. Facts about the request travel with the request (ZFN-51Field Note · currentZFN-51 — Design the request envelope before the first endpointAuth, idempotency keys, trace IDs, vector clocks: context about a request, not part of it. Define an envelope alongside the payload in the schema on day one — transport-native, per-hop vs propagated, owned by generated SDKs and middleware. The retrofit is what costs you.Why it's cited here: Where the deadline physically rides: an envelope field defined once, carried by every hop, decremented in middleware — never a body field re-invented per endpoint.Open ZFN-51 →). Once it travels, every hop can make the locally-correct decision — do the work, shed it, or stop it mid-flight — and the per-hop guesses become what they should have been: a backstop for peers that don’t speak the protocol, not the design.

One boundary needs marking: this is a synchronous-path discipline. A queued job whose requester stopped waiting may still be worth running — async work’s contract is durability, not immediacy (ZFN-12Field Note · currentZFN-12 — Queues, topics, and journals are different tools — don't conflate themQueues (competing consumers), topics (fan-out), and journals (ordered, replayable logs) give different guarantees. Don't conflate them; a pipeline often uses several. Prefer journals over topics, but not where head-of-line blocking hurts. With queues, bound the concurrency.Open ZFN-12 →); its runaway risk is bounded by budgets of its own (ZFN-39Field Note · currentZFN-39 — Break loops, not spiralsEvent-driven and agentic systems echo. A loop — a lineage recurring with the same data — produces nothing new; detect it with runtime-stamped provenance and break it loudly. A spiral — recurring with new data — is legitimate work; bound it with budgets, never loop-breakers.Open ZFN-39 →). What crosses into the queue is not the RPC deadline but a freshness bound where one exists — “don’t bother sending this notification after 10 minutes” is the job’s own fact, minted at enqueue.

Recommendation

Make the deadline a first-class field, then spend it like the budget it is.

• Mint at the edge, from the product. Interactive requests get what a human will wait — single-digit seconds. Batch-facing APIs get what the workflow tolerates. The number is a product decision made once per entry point, versioned in the gateway config — not two hundred scattered client defaults.

• Wire format: remaining duration, decremented per hop — each hop subtracts its own processing time before calling on. An absolute timestamp is cleaner arithmetic and a cross-machine clock dependency you must not take (ZFN-59Field Note · currentZFN-59 — Two clocks: monotonic for durations, wall time for recordsYou have two clocks. Wall time names moments — and it jumps, slews, and runs backwards. The monotonic clock measures elapsed time — and means nothing across machines or reboots. Every timeout, lease, and cross-machine ordering bug is one clock doing the other's job.Why it's cited here: Why the wire format is remaining-duration rather than an absolute instant: an absolute deadline silently requires every machine's wall clock to agree.Open ZFN-59 →); measure the decrement with the monotonic clock. Cap what you’ll honour at ingress: a caller presenting a one-hour deadline is asking to occupy your capacity — the cap is a quota decision (ZFN-18Field Note · currentZFN-18 — Enforce a quota at ingress on every endpoint — even unabused onesPut a quota on every endpoint and enforce it at ingress from day one — per tenant, principal, IP — even for endpoints nobody abuses yet. Unlimited-by-default means the first runaway client or compromised key is an outage. Return 429 + Retry-After; retrofitting limits is painful.Open ZFN-18 →).

• Check before spending, at the natural toll gates — middleware on entry (a request that queued past its budget dies before the handler: fail fast where the work hasn’t happened yet, ZFN-13Field Note · currentZFN-13 — Fail fast and push back: retries, load shedding, and flow controlBuild client retries (backoff, jitter, Retry-After) from day one. Under overload, shed fast and push the failure back to the source to retry — don't retry internally and amplify it. Flow-control everywhere, bound every queue, and don't take more work than you can finish in time.Why it's cited here: The companion discipline: this note stops you doing work nobody is waiting for; that one stops you accepting work you can't finish in time.Open ZFN-13 →), before expensive steps, and at every downstream call (pass the remainder; if it’s below the callee’s realistic floor, don’t make the call). Set the per-hop timeout from the propagated remainder — the local guess becomes the fallback, not the truth.

• Cancellation is the other half. When the caller disconnects or the budget dies, propagate cancellation down the tree — modern stacks carry a context/abort signal for exactly this; the work is honouring it in handlers and queries (databases can kill a running query; most services never ask). deadline_exceeded is a distinct, terminal, non-retryable error code (ZFN-58Field Note · currentZFN-58 — Errors are part of the contractError paths are the half of your API clients depend on most, and usually the half nobody designed. Enumerate error codes in the schema like any other type: stable code, retryable-or-not, whose fault, structured params. Machines branch on codes — anyone parsing prose is broken.Open ZFN-58 →): retrying it against the same budget is definitionally pointless.

• Retries live inside the budget. The retry policy and the deadline are one design: each attempt gets a slice, backoff between slices, and the whole envelope closes when the budget does. Retry budgets per client-server pair cap amplification fleet-wide (the SRE mechanismsAddressing Cascading Failures (Google SRE Book)Google's field guide to overload collapse, including the specific mechanisms this note leans on: deadline propagation to stop servers doing work no caller is waiting for, retry budgets to stop amplification, and load shedding to keep goodput up when demand exceeds capacity.sre.google ↗ again). Hedged requests, if you use them, also spend from the same allowance.

• Middleware and generated clients own the mechanics (ZFN-51Field Note · currentZFN-51 — Design the request envelope before the first endpointAuth, idempotency keys, trace IDs, vector clocks: context about a request, not part of it. Define an envelope alongside the payload in the schema on day one — transport-native, per-hop vs propagated, owned by generated SDKs and middleware. The retrofit is what costs you.Why it's cited here: Where the deadline physically rides: an envelope field defined once, carried by every hop, decremented in middleware — never a body field re-invented per endpoint.Open ZFN-51 →, ZFN-14Field Note · currentZFN-14 — Define every API with a schema, and generate the clientsDefine every API with a machine-readable schema (OpenAPI, Protobuf, GraphQL) as the source of truth, and generate clients and server stubs from it — never hand-roll request-building and JSON parsing. Hand-written clients drift and break silently; check schema compatibility in CI.Open ZFN-14 →): decrement, check, propagate, cancel — application code reads the remaining budget only when it can genuinely adapt (return partial results, skip the enrichment, choose the cheaper path). That adaptive move is where the propagated deadline turns from a reliability mechanism into a product feature: graceful degradation with an actual number to degrade against.

Consequences

Easier:

• Overload stops compounding. Abandoned work is cancelled instead of completed, so capacity flows to requests someone still wants — the difference between a latency blip and an afternoon-long brownout.
• End-to-end latency has a contract. “This endpoint’s budget is 2s” is a reviewable, testable statement, and every hop’s share of it is visible in traces.
• Incident behaviour gets legible: budgets exhausted at hop three point at hop three; before, the same failure surfaced as mysterious timeouts at every layer at once.

Harder:

• Every hop must play, or the chain breaks silently — the proxy that drops the header, the legacy service that ignores it, the language client with no cancellation story. Partial propagation degrades soft (inner hops fall back to local timeouts), but audit where the chain actually ends; the envelope checklist (ZFN-51Field Note · currentZFN-51 — Design the request envelope before the first endpointAuth, idempotency keys, trace IDs, vector clocks: context about a request, not part of it. Define an envelope alongside the payload in the schema on day one — transport-native, per-hop vs propagated, owned by generated SDKs and middleware. The retrofit is what costs you.Why it's cited here: Where the deadline physically rides: an envelope field defined once, carried by every hop, decremented in middleware — never a body field re-invented per endpoint.Open ZFN-51 →) is where that lives.
• Cancellation correctness is real work: a mutation killed mid-flight must be safe to retry or abandon — idempotency (ZFN-19Field Note · currentZFN-19 — Annotate read-only and idempotent endpoints; make every mutation idempotentAnnotate every endpoint as read-only (safe) or idempotent, in the schema, so infrastructure can retry, route to replicas, and cache safely. Make every state-changing endpoint idempotent (idempotency keys for create/charge/send); a non-idempotent retry double-applies.Open ZFN-19 →) and fencing (ZFN-37Field Note · currentZFN-37 — Every lock is a leaseA lock that can outlive its holder is a deadlock scheduled for later. Give every lock — including informal ones like claimed_by columns — a TTL, a named owner, and a heartbeat; make expiry automatic and server-side; fence the side effects so a stale holder can't corrupt anything.Open ZFN-37 →) are prerequisites, not nice-to-haves, before you cancel aggressively around writes.
• Edge budgets force uncomfortable conversations — the report that legitimately needs 40 seconds can’t hide inside a generous default timeout anymore; it has to become an async job with a status endpoint, which is the correct answer someone now has to build (ZFN-12Field Note · currentZFN-12 — Queues, topics, and journals are different tools — don't conflate themQueues (competing consumers), topics (fan-out), and journals (ordered, replayable logs) give different guarantees. Don't conflate them; a pipeline often uses several. Prefer journals over topics, but not where head-of-line blocking hurts. With queues, bound the concurrency.Open ZFN-12 →).

References

• ZFN-13Field Note · currentZFN-13 — Fail fast and push back: retries, load shedding, and flow controlBuild client retries (backoff, jitter, Retry-After) from day one. Under overload, shed fast and push the failure back to the source to retry — don't retry internally and amplify it. Flow-control everywhere, bound every queue, and don't take more work than you can finish in time.Why it's cited here: The companion discipline: this note stops you doing work nobody is waiting for; that one stops you accepting work you can't finish in time.Open ZFN-13 → — the intake half: shed what you can’t finish; this note is the spend half: stop what nobody wants.
• ZFN-51Field Note · currentZFN-51 — Design the request envelope before the first endpointAuth, idempotency keys, trace IDs, vector clocks: context about a request, not part of it. Define an envelope alongside the payload in the schema on day one — transport-native, per-hop vs propagated, owned by generated SDKs and middleware. The retrofit is what costs you.Why it's cited here: Where the deadline physically rides: an envelope field defined once, carried by every hop, decremented in middleware — never a body field re-invented per endpoint.Open ZFN-51 → — the deadline is the canonical envelope field, owned by middleware and generated clients.
• ZFN-59Field Note · currentZFN-59 — Two clocks: monotonic for durations, wall time for recordsYou have two clocks. Wall time names moments — and it jumps, slews, and runs backwards. The monotonic clock measures elapsed time — and means nothing across machines or reboots. Every timeout, lease, and cross-machine ordering bug is one clock doing the other's job.Why it's cited here: Why the wire format is remaining-duration rather than an absolute instant: an absolute deadline silently requires every machine's wall clock to agree.Open ZFN-59 → — remaining-duration on the wire because absolute instants smuggle in a clock-agreement dependency.
• ZFN-58Field Note · currentZFN-58 — Errors are part of the contractError paths are the half of your API clients depend on most, and usually the half nobody designed. Enumerate error codes in the schema like any other type: stable code, retryable-or-not, whose fault, structured params. Machines branch on codes — anyone parsing prose is broken.Open ZFN-58 → — deadline_exceeded as a first-class, terminal error code.
• gRPC and DeadlinesgRPC and Deadlines (gRPC blog)The practitioner's introduction to deadline propagation as a first-class RPC feature: callers set a deadline, every hop inherits what remains, servers check it before doing work, and cancellation flows downstream when the deadline expires or the caller goes away.grpc.io ↗ — the mechanism, productised; Addressing Cascading FailuresAddressing Cascading Failures (Google SRE Book)Google's field guide to overload collapse, including the specific mechanisms this note leans on: deadline propagation to stop servers doing work no caller is waiting for, retry budgets to stop amplification, and load shedding to keep goodput up when demand exceeds capacity.sre.google ↗ — why systems without it stay down.

Changelog

• 2026-08-12: First published as a Field Note.