--- id: 61 title: "Propagate the deadline" kind: note status: current date: 2026-08-12 authors: - "Theo Zourzouvillys" tags: [reliability, architecture, api, resilience] references: - id: grpcdeadlines title: "gRPC and Deadlines (gRPC blog)" url: https://grpc.io/blog/deadlines/ abstract: "The practitioner's introduction to deadline propagation as a first-class RPC feature: callers set a deadline, every hop inherits what remains, servers check it before doing work, and cancellation flows downstream when the deadline expires or the caller goes away." - id: srecascading title: "Addressing Cascading Failures (Google SRE Book)" url: https://sre.google/sre-book/addressing-cascading-failures/ abstract: "Google's field guide to overload collapse, including the specific mechanisms this note leans on: deadline propagation to stop servers doing work no caller is waiting for, retry budgets to stop amplification, and load shedding to keep goodput up when demand exceeds capacity." summary: "Every request has a deadline whether you set one or not — the caller's patience. Make it explicit at the edge, carry it as remaining budget on every hop, check it before expensive steps, and cancel downstream when it dies. Work past the deadline is the fuel of cascading collapse." supersedes: null superseded_by: null aliases: [] crossrefs: ZFN-13: "The companion discipline: this note stops you doing work nobody is waiting for; that one stops you accepting work you can't finish in time." ZFN-51: "Where the deadline physically rides: an envelope field defined once, carried by every hop, decremented in middleware — never a body field re-invented per endpoint." ZFN-59: "Why the wire format is remaining-duration rather than an absolute instant: an absolute deadline silently requires every machine's wall clock to agree." --- ## TL;DR **Every request already has a deadline. The only question is whether your system knows it.** The user gives up, the upstream times out, the load balancer cuts the connection — after that moment, every cycle spent on the request is pure waste, and under load it's worse than waste: [servers grinding through work no one is waiting for is the signature fuel of cascading collapse](ref:srecascading). So make the implicit explicit: - **Set the deadline once, at the edge**, from product truth — how long is this interaction worth to a human or a calling system? — not from per-hop latency folklore. - **Carry it on every hop** as an envelope field ([ZFN-51](/zfn/51-design-the-request-envelope-first/)), as **remaining budget** (a duration, decremented per hop) rather than an absolute timestamp, so no two machines' clocks need to agree ([ZFN-59](/zfn/59-two-clocks/)). - **Check it before spending**: on dequeue, before the expensive query, before the downstream call. Not enough budget left for the work? **Fail now, cheaply** — a deadline-exceeded error ([ZFN-58](/zfn/58-errors-are-part-of-the-contract/)) beats a timeout later, by exactly the resources you didn't burn. - **Cancel downstream when the caller is gone.** A dead deadline should ripple down the call tree and stop work everywhere ([gRPC ships all of this as a built-in](ref:grpcdeadlines); HTTP stacks make you assemble it). - **Fit retries inside the deadline, never on top of it** ([ZFN-13](/zfn/13-load-shedding-and-flow-control/)). ## Context The default, in almost every HTTP shop, is per-hop timeouts chosen independently: the gateway gives 30s, the service calls the next with its client default of 60, which calls the database with 30 more. Three consequences, all bad: - **Inner hops outlive outer ones.** The gateway gave up at 30s; the database is still faithfully executing the query at minute two. Multiply by every request during an incident and the database is spending its capacity almost exclusively on abandoned work — which is why the system *stays* down after the triggering blip has passed ([the SRE book's cascading-failure chapter](ref:srecascading) is substantially a catalogue of this one mistake). - **Nobody can answer "how long may this take?"** The end-to-end budget is an emergent property of scattered config values, discovered empirically, during incidents. - **Timeouts stack with retries multiplicatively.** Three tiers each retrying three times with independent timeouts is up to 27 attempts against the struggling dependency — retry amplification wired in at design time. The reframe: a **timeout** is a local, defensive guess; a **deadline** is a fact about the request — *after instant T, the answer has no value*. Facts about the request travel with the request ([ZFN-51](/zfn/51-design-the-request-envelope-first/)). Once it travels, every hop can make the locally-correct decision — do the work, shed it, or stop it mid-flight — and the per-hop guesses become what they should have been: a backstop for peers that don't speak the protocol, not the design. One boundary needs marking: **this is a synchronous-path discipline.** A queued job whose *requester* stopped waiting may still be worth running — async work's contract is durability, not immediacy ([ZFN-12](/zfn/12-queues-topics-journals/)); its runaway risk is bounded by budgets of its own ([ZFN-39](/zfn/39-break-loops-not-spirals/)). What crosses into the queue is not the RPC deadline but a *freshness* bound where one exists — "don't bother sending this notification after 10 minutes" is the job's own fact, minted at enqueue. > [!aside] > > The cheapest observability win in this whole area: log remaining budget at each hop, and > alert on work that *completed* after its deadline expired. That counter is invisible in > normal dashboards — the response was thrown away, so nothing errored — but it is precisely > the fuel gauge of your next cascading failure, readable weeks in advance. ## Recommendation **Make the deadline a first-class field, then spend it like the budget it is.** - **Mint at the edge, from the product.** Interactive requests get what a human will wait — single-digit seconds. Batch-facing APIs get what the workflow tolerates. The number is a product decision made once per entry point, versioned in the gateway config — not two hundred scattered client defaults. - **Wire format: remaining duration, decremented per hop** — each hop subtracts its own processing time before calling on. An absolute timestamp is cleaner arithmetic and a cross-machine clock dependency you must not take ([ZFN-59](/zfn/59-two-clocks/)); measure the decrement with the monotonic clock. Cap what you'll honour at ingress: a caller presenting a one-hour deadline is asking to occupy your capacity — the cap is a quota decision ([ZFN-18](/zfn/18-enforce-quotas-at-ingress/)). - **Check before spending, at the natural toll gates** — middleware on entry (a request that queued past its budget dies *before* the handler: fail fast where the work hasn't happened yet, [ZFN-13](/zfn/13-load-shedding-and-flow-control/)), before expensive steps, and at every downstream call (pass the remainder; if it's below the callee's realistic floor, don't make the call). Set the per-hop timeout *from* the propagated remainder — the local guess becomes the fallback, not the truth. - **Cancellation is the other half.** When the caller disconnects or the budget dies, propagate cancellation down the tree — modern stacks carry a context/abort signal for exactly this; the work is honouring it in handlers and queries (databases can kill a running query; most services never ask). `deadline_exceeded` is a distinct, terminal, non-retryable error code ([ZFN-58](/zfn/58-errors-are-part-of-the-contract/)): retrying it against the same budget is definitionally pointless. - **Retries live inside the budget.** The retry policy and the deadline are one design: each attempt gets a slice, backoff between slices, and the whole envelope closes when the budget does. Retry budgets per client-server pair cap amplification fleet-wide ([the SRE mechanisms](ref:srecascading) again). Hedged requests, if you use them, also spend from the same allowance. - **Middleware and generated clients own the mechanics** ([ZFN-51](/zfn/51-design-the-request-envelope-first/), [ZFN-14](/zfn/14-schema-first-apis-generate-clients/)): decrement, check, propagate, cancel — application code reads the remaining budget only when it can genuinely adapt (return partial results, skip the enrichment, choose the cheaper path). That adaptive move is where the propagated deadline turns from a reliability mechanism into a product feature: graceful degradation with an actual number to degrade against. ## Consequences **Easier:** - **Overload stops compounding.** Abandoned work is cancelled instead of completed, so capacity flows to requests someone still wants — the difference between a latency blip and an afternoon-long brownout. - **End-to-end latency has a contract.** "This endpoint's budget is 2s" is a reviewable, testable statement, and every hop's share of it is visible in traces. - **Incident behaviour gets legible**: budgets exhausted at hop three point at hop three; before, the same failure surfaced as mysterious timeouts at every layer at once. **Harder:** - **Every hop must play, or the chain breaks silently** — the proxy that drops the header, the legacy service that ignores it, the language client with no cancellation story. Partial propagation degrades soft (inner hops fall back to local timeouts), but audit where the chain actually ends; the envelope checklist ([ZFN-51](/zfn/51-design-the-request-envelope-first/)) is where that lives. - **Cancellation correctness is real work**: a mutation killed mid-flight must be safe to retry or abandon — idempotency ([ZFN-19](/zfn/19-annotate-readonly-idempotent-endpoints/)) and fencing ([ZFN-37](/zfn/37-every-lock-is-a-lease/)) are prerequisites, not nice-to-haves, before you cancel aggressively around writes. - **Edge budgets force uncomfortable conversations** — the report that legitimately needs 40 seconds can't hide inside a generous default timeout anymore; it has to become an async job with a status endpoint, which is the correct answer someone now has to build ([ZFN-12](/zfn/12-queues-topics-journals/)). ## References - [ZFN-13](/zfn/13-load-shedding-and-flow-control/) — the intake half: shed what you can't finish; this note is the spend half: stop what nobody wants. - [ZFN-51](/zfn/51-design-the-request-envelope-first/) — the deadline is the canonical envelope field, owned by middleware and generated clients. - [ZFN-59](/zfn/59-two-clocks/) — remaining-duration on the wire because absolute instants smuggle in a clock-agreement dependency. - [ZFN-58](/zfn/58-errors-are-part-of-the-contract/) — `deadline_exceeded` as a first-class, terminal error code. - [gRPC and Deadlines](ref:grpcdeadlines) — the mechanism, productised; [Addressing Cascading Failures](ref:srecascading) — why systems without it stay down. ## Changelog - **2026-08-12**: First published as a Field Note.