Drain before you die: graceful shutdown is a protocol

Context

Watch a team’s error dashboard during deploys: a small spike of 502s and connection resets on every rollout, small enough to ignore, normalised until nobody sees it. That spike is the gap between two views of the world — the process knows it’s dying; the load balancer hasn’t heard yet. It keeps routing; the process has closed its listener; connections land on a corpse.

The failure is architectural, not accidental: shutdown is a distributed handoff being treated as a local event. The pieces that must agree — orchestrator, balancer, the process, its queue brokers, its lease holders — learn the news at different times, and the protocol above is nothing but choreography for that propagation delay. Its details are unforgiving:

• Readiness and liveness are different questions with different consequences. “Don’t send me new work” (readiness, fail it early and deliberately) versus “I’m wedged, kill me” (liveness — keep passing it while draining, or the platform helpfully converts the graceful path into the kill you were avoiding).
• In-flight is more than open sockets: queue messages mid-lease, half-applied batch items, a WebSocket per customer, cron work started ten seconds ago. Each needs an owner-decided fate — finish, checkpoint, or hand back — inside the budget.
• The budget is hierarchical. Thirty seconds total means the HTTP drain gets ten, the queue handoff gets ten, the flush gets five, and the slowest single request you’re willing to wait for is bounded by the first number — which is a product decision about the longest request you should be serving at all (ZFN-61Field Note · currentZFN-61 — Propagate the deadlineEvery request has a deadline whether you set one or not — the caller's patience. Make it explicit at the edge, carry it as remaining budget on every hop, check it before expensive steps, and cancel downstream when it dies. Work past the deadline is the fuel of cascading collapse.Open ZFN-61 →).

The deeper trap is the one Candea and FoxCrash-Only Software (Candea & Fox, HotOS 2003)The position paper arguing that stop = crash and start = recover should be the only code paths: software that is always safe to kill and always starts by recovering is simpler and more reliable than software with a separate, rarely-exercised clean-shutdown path it secretly depends on.usenix.org ↗ named: a clean-shutdown path that correctness quietly starts to depend on. The flush-on-exit that’s the only thing writing the buffer out; the “save state on SIGTERM” that’s the only persistence; the lock released only in the shutdown hook. Every one of those is a bug with a delay on it, because the one guarantee about SIGKILL is that it’s coming — eventually, uninvited, mid-write.

Recommendation

Design the crash first, then add the grace.

• Make unclean death a non-event before optimising clean death. Every mutation idempotent or fenced (ZFN-19Field Note · currentZFN-19 — Annotate read-only and idempotent endpoints; make every mutation idempotentAnnotate every endpoint as read-only (safe) or idempotent, in the schema, so infrastructure can retry, route to replicas, and cache safely. Make every state-changing endpoint idempotent (idempotency keys for create/charge/send); a non-idempotent retry double-applies.Why it's cited here: Idempotency is what makes the hand-back safe: work interrupted mid-flight gets retried by someone else, and only annotated-idempotent operations make that a non-event.Open ZFN-19 →, ZFN-37Field Note · currentZFN-37 — Every lock is a leaseA lock that can outlive its holder is a deadlock scheduled for later. Give every lock — including informal ones like claimed_by columns — a TTL, a named owner, and a heartbeat; make expiry automatic and server-side; fence the side effects so a stale holder can't corrupt anything.Why it's cited here: What happens to held leases at shutdown is the test of the lease design: release them if you can, and rely on expiry plus fencing when you can't — because SIGKILL releases nothing.Open ZFN-37 →), every queue message redelivered on lease expiry, every durable fact in the store — never only in memory en route to a shutdown hook (ZFN-24Field Note · currentZFN-24 — One transactional store per write; propagate changes asynchronouslyCommit each logical write to exactly one transactional store; update other systems via reliable ordered async events — never a synchronous write across two stores, and never 2PC. With a relational primary the WAL is your replayable journal; write events into the same transaction.Open ZFN-24 →). The test is literal: kill -9 under load must cost latency, not correctness.

• Then implement the drain, in order: trap SIGTERM; fail readiness immediately while serving on; give the balancer its convergence window (a few seconds of overlap — the platform documents the numberGraceful shutdowns with ECS (AWS Containers blog)AWS's worked description of the ECS termination sequence — deregistration from the load balancer, connection draining, SIGTERM, the stopTimeout window, then SIGKILL — and what a task must do at each step to exit without dropping traffic.aws.amazon.com ↗); stop intake everywhere (listeners, queue consumers, schedulers — the queue pollers are the ones everyone forgets); bound the wait for in-flight work; nack or checkpoint what won’t make it; release leases explicitly so successors start now rather than at TTL expiry; flush spans and metrics; exit 0.

• Know your platform’s actual numbers and budget inside them. Grace periods, deregistration delays, connection-drain settings — read them, set them deliberately, and alert when drains overrun the budget: an overrunning drain is either a too-slow endpoint or a too-small grace, and both are findable in daylight.

• Long-lived connections get a protocol of their own. WebSockets and streams can’t “finish” — send GOAWAY or a reconnect hint, let clients re-establish against live instances (ZFN-13Field Note · currentZFN-13 — Fail fast and push back: retries, load shedding, and flow controlBuild client retries (backoff, jitter, Retry-After) from day one. Under overload, shed fast and push the failure back to the source to retry — don't retry internally and amplify it. Flow-control everywhere, bound every queue, and don't take more work than you can finish in time.Why it's cited here: Draining is flow control pointed at yourself: stop taking work you cannot finish inside the budget, and push it back to a peer who can.Open ZFN-13 →: the reconnect storm is load you’re shedding onto your own fleet — pace the drain).

• One implementation, in the platform layer. Shutdown choreography is exactly the cross-cutting machinery that belongs in the shared runtime scaffolding every service gets (ZFN-47Field Note · currentZFN-47 — Govern the contract between teams, not the code inside themTeams own services end to end; one team owns the gateway that dispatches to them. Govern exactly one thing centrally — the contract at the boundary (schema, identity, errors, idempotency) — and enforce it at runtime. Don't mandate libraries; ship them as an opt-in blueprint.Open ZFN-47 →), not two hundred lines of signal-handling folklore re-derived per service. It’s also where the order is enforced — the classic self-inflicted outage is a service that closes its database pool before its HTTP drain finishes, gracefully serving 500s to every request it gracefully accepted.

Consequences

Easier:

• Deploys stop having an error budget cost, which is what makes deploying often politically free — the reliability argument for shipping small (ZFN-23Field Note · currentZFN-23 — Rewriting an implementation is fine — refactoring isn't always the answerRefactoring isn't always right. When the structure is wrong at the root, it's fine — often better — to rewrite an implementation from scratch. Clean interfaces and data models make the implementation disposable: stable contract, swappable internals. LLMs make it cheaper still.Open ZFN-23 → gets cheaper when rollout is invisible).
• Autoscaling and spot capacity become usable aggressively — scale-in is safe at any hour, and the spot discount stops costing correctness.
• Handovers get fast: explicit lease release and queue nacks mean successors take over in milliseconds instead of waiting out TTLs.

Harder:

• The drain path is real code with real ordering constraints, and it’s exercised constantly — which is the good news wearing work clothes: bugs in it surface in daylight deploys, not 3 a.m. failovers.
• Budgeting forces uncomfortable honesty about your longest requests and slowest jobs; the request that can’t finish inside any reasonable grace period was always a problem — shutdown design is merely where it stops being deniable.
• Two disciplines, permanently: the crash-safe substrate and the graceful layer, with the standing temptation to let the second quietly excuse gaps in the first. The kill -9 habit is the immune system for exactly that drift.

References

• ZFN-37Field Note · currentZFN-37 — Every lock is a leaseA lock that can outlive its holder is a deadlock scheduled for later. Give every lock — including informal ones like claimed_by columns — a TTL, a named owner, and a heartbeat; make expiry automatic and server-side; fence the side effects so a stale holder can't corrupt anything.Why it's cited here: What happens to held leases at shutdown is the test of the lease design: release them if you can, and rely on expiry plus fencing when you can't — because SIGKILL releases nothing.Open ZFN-37 → — leases, expiry, and fencing: the machinery that makes both clean and unclean death safe for held work.
• ZFN-19Field Note · currentZFN-19 — Annotate read-only and idempotent endpoints; make every mutation idempotentAnnotate every endpoint as read-only (safe) or idempotent, in the schema, so infrastructure can retry, route to replicas, and cache safely. Make every state-changing endpoint idempotent (idempotency keys for create/charge/send); a non-idempotent retry double-applies.Why it's cited here: Idempotency is what makes the hand-back safe: work interrupted mid-flight gets retried by someone else, and only annotated-idempotent operations make that a non-event.Open ZFN-19 → — idempotency, which turns “interrupted and retried” into a non-event.
• ZFN-13Field Note · currentZFN-13 — Fail fast and push back: retries, load shedding, and flow controlBuild client retries (backoff, jitter, Retry-After) from day one. Under overload, shed fast and push the failure back to the source to retry — don't retry internally and amplify it. Flow-control everywhere, bound every queue, and don't take more work than you can finish in time.Why it's cited here: Draining is flow control pointed at yourself: stop taking work you cannot finish inside the budget, and push it back to a peer who can.Open ZFN-13 → — draining as flow control, including the reconnect storm you create by draining carelessly.
• ZFN-24Field Note · currentZFN-24 — One transactional store per write; propagate changes asynchronouslyCommit each logical write to exactly one transactional store; update other systems via reliable ordered async events — never a synchronous write across two stores, and never 2PC. With a relational primary the WAL is your replayable journal; write events into the same transaction.Open ZFN-24 → — durable facts live in the store, never in a buffer waiting on a shutdown hook.
• Crash-Only SoftwareCrash-Only Software (Candea & Fox, HotOS 2003)The position paper arguing that stop = crash and start = recover should be the only code paths: software that is always safe to kill and always starts by recovering is simpler and more reliable than software with a separate, rarely-exercised clean-shutdown path it secretly depends on.usenix.org ↗ — the argument that stop=crash is the only shutdown contract you can trust; ECSGraceful shutdowns with ECS (AWS Containers blog)AWS's worked description of the ECS termination sequence — deregistration from the load balancer, connection draining, SIGTERM, the stopTimeout window, then SIGKILL — and what a task must do at each step to exit without dropping traffic.aws.amazon.com ↗ and KubernetesPod Lifecycle: Termination of Pods (Kubernetes documentation)The other major orchestrator's statement of the same contract: SIGTERM plus a grace period (default 30 seconds) then SIGKILL, with endpoint removal proceeding in parallel with — not before — signal delivery, which is why draining must overlap serving.kubernetes.io ↗ — the concrete choreography and the numbers to budget inside.

Changelog

• 2026-08-12: First published as a Field Note.