---
id: 60
title: "Drain before you die: graceful shutdown is a protocol"
kind: note
status: current
date: 2026-08-12
authors:
  - "Theo Zourzouvillys"
tags: [reliability, operations, infra, architecture]
references:
  - id: crashonly
    title: "Crash-Only Software (Candea & Fox, HotOS 2003)"
    url: https://www.usenix.org/legacy/events/hotos03/tech/full_papers/candea/candea.pdf
    abstract: "The position paper arguing that stop = crash and start = recover should be the only code paths: software that is always safe to kill and always starts by recovering is simpler and more reliable than software with a separate, rarely-exercised clean-shutdown path it secretly depends on."
  - id: ecsshutdown
    title: "Graceful shutdowns with ECS (AWS Containers blog)"
    url: https://aws.amazon.com/blogs/containers/graceful-shutdowns-with-ecs/
    abstract: "AWS's worked description of the ECS termination sequence — deregistration from the load balancer, connection draining, SIGTERM, the stopTimeout window, then SIGKILL — and what a task must do at each step to exit without dropping traffic."
  - id: k8stermination
    title: "Pod Lifecycle: Termination of Pods (Kubernetes documentation)"
    url: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination
    abstract: "The other major orchestrator's statement of the same contract: SIGTERM plus a grace period (default 30 seconds) then SIGKILL, with endpoint removal proceeding in parallel with — not before — signal delivery, which is why draining must overlap serving."
summary: "SIGTERM isn't an emergency — it's every deploy and scale-in. Shutdown is a protocol: stop attracting work, drain while the balancer catches up, hand back in-flight work, release leases, exit before SIGKILL. But graceful is only the optimisation — crash-safe is the requirement."
supersedes: null
superseded_by: null
aliases: []
crossrefs:
  ZFN-37: "What happens to held leases at shutdown is the test of the lease design: release them if you can, and rely on expiry plus fencing when you can't — because SIGKILL releases nothing."
  ZFN-19: "Idempotency is what makes the hand-back safe: work interrupted mid-flight gets retried by someone else, and only annotated-idempotent operations make that a non-event."
  ZFN-13: "Draining is flow control pointed at yourself: stop taking work you cannot finish inside the budget, and push it back to a peer who can."
---

## TL;DR

**Termination is not an exceptional event. It's every deploy, every scale-in, every spot
reclaim — for a healthy service, shutting down is one of the most frequently executed paths in
the system, and one of the least designed.** Treat it as a protocol with ordered steps and a
hard deadline:

1. **On SIGTERM, stop attracting work** — fail readiness, deregister — while continuing to
   serve: the balancer's view of you converges asynchronously, and the requests that arrive in
   that gap are yours ([ECS](ref:ecsshutdown) and [Kubernetes](ref:k8stermination) both deliver
   the signal *in parallel* with deregistration, not after it).
2. **Then stop intake**: close listeners, stop polling queues, cancel timers that start new
   work.
3. **Finish or hand back what's in flight** within the remaining budget — complete the fast,
   checkpoint or nack the slow ([ZFN-12](/zfn/12-queues-topics-journals/)), release held leases
   ([ZFN-37](/zfn/37-every-lock-is-a-lease/)).
4. **Flush telemetry** — the spans from the drain are the ones you'll want in the postmortem.
5. **Exit zero, before the platform's SIGKILL deadline** — 30 seconds by default on both major
   orchestrators. Everything above must be budgeted inside it.

And underneath all of it, the [crash-only](ref:crashonly) rule: **the graceful path is an
optimisation, never a correctness mechanism.** SIGKILL, OOM, and hardware failure skip the
protocol entirely, so recovery — idempotent retries
([ZFN-19](/zfn/19-annotate-readonly-idempotent-endpoints/)), lease expiry with fencing, journal
replay — must make an unclean death survivable. Graceful shutdown exists to make routine
deaths *invisible*, not possible.

## Context

Watch a team's error dashboard during deploys: a small spike of 502s and connection resets on
every rollout, small enough to ignore, normalised until nobody sees it. That spike is the gap
between two views of the world — the process knows it's dying; the load balancer hasn't heard
yet. It keeps routing; the process has closed its listener; connections land on a corpse.

The failure is architectural, not accidental: **shutdown is a distributed handoff being treated
as a local event.** The pieces that must agree — orchestrator, balancer, the process, its queue
brokers, its lease holders — learn the news at different times, and the protocol above is
nothing but choreography for that propagation delay. Its details are unforgiving:

- **Readiness and liveness are different questions with different consequences.** "Don't send
  me new work" (readiness, fail it early and deliberately) versus "I'm wedged, kill me"
  (liveness — keep passing it while draining, or the platform helpfully converts the graceful
  path into the kill you were avoiding).
- **In-flight is more than open sockets**: queue messages mid-lease, half-applied batch items,
  a WebSocket per customer, cron work started ten seconds ago. Each needs an owner-decided
  fate — finish, checkpoint, or hand back — inside the budget.
- **The budget is hierarchical.** Thirty seconds total means the HTTP drain gets ten, the queue
  handoff gets ten, the flush gets five, and the slowest single request you're willing to wait
  for is bounded by the first number — which is a *product* decision about the longest request
  you should be serving at all ([ZFN-61](/zfn/61-propagate-the-deadline/)).

The deeper trap is the one [Candea and Fox](ref:crashonly) named: a clean-shutdown path that
correctness quietly starts to *depend on*. The flush-on-exit that's the only thing writing the
buffer out; the "save state on SIGTERM" that's the only persistence; the lock released only in
the shutdown hook. Every one of those is a bug with a delay on it, because the one guarantee
about SIGKILL is that it's coming — eventually, uninvited, mid-write.

> [!aside]
>
> A habit worth stealing from chaos engineering, at zero tooling cost: `kill -9` one instance
> in staging as part of the deploy pipeline, every deploy. Not to test the graceful path — to
> prove you never needed it. The teams that do this stop fearing deploys; the teams that fear
> deploys have, without writing it down anywhere, made clean shutdown a correctness
> requirement.

## Recommendation

**Design the crash first, then add the grace.**

- **Make unclean death a non-event before optimising clean death.** Every mutation idempotent
  or fenced ([ZFN-19](/zfn/19-annotate-readonly-idempotent-endpoints/),
  [ZFN-37](/zfn/37-every-lock-is-a-lease/)), every queue message redelivered on lease expiry,
  every durable fact in the store — never only in memory en route to a shutdown hook
  ([ZFN-24](/zfn/24-one-transactional-store-per-write/)). The test is literal: `kill -9` under
  load must cost latency, not correctness.

- **Then implement the drain, in order:** trap SIGTERM; fail readiness immediately while
  serving on; give the balancer its convergence window (a few seconds of overlap — the
  [platform documents the number](ref:ecsshutdown)); stop intake everywhere (listeners, queue
  consumers, schedulers — the queue pollers are the ones everyone forgets); bound the wait for
  in-flight work; nack or checkpoint what won't make it; release leases explicitly so
  successors start now rather than at TTL expiry; flush spans and metrics; exit 0.

- **Know your platform's actual numbers and budget inside them.** Grace periods, deregistration
  delays, connection-drain settings — read them, set them deliberately, and alert when drains
  overrun the budget: an overrunning drain is either a too-slow endpoint or a too-small grace,
  and both are findable in daylight.

- **Long-lived connections get a protocol of their own.** WebSockets and streams can't
  "finish" — send GOAWAY or a reconnect hint, let clients re-establish against live instances
  ([ZFN-13](/zfn/13-load-shedding-and-flow-control/): the reconnect storm is load you're
  shedding onto your own fleet — pace the drain).

- **One implementation, in the platform layer.** Shutdown choreography is exactly the
  cross-cutting machinery that belongs in the shared runtime scaffolding every service gets
  ([ZFN-47](/zfn/47-govern-the-contract-between-teams/)), not two hundred lines of
  signal-handling folklore re-derived per service. It's also where the *order* is enforced —
  the classic self-inflicted outage is a service that closes its database pool before its HTTP
  drain finishes, gracefully serving 500s to every request it gracefully accepted.

## Consequences

**Easier:**

- **Deploys stop having an error budget cost**, which is what makes deploying often politically
  free — the reliability argument for shipping small
  ([ZFN-23](/zfn/23-iterate-and-rewrite-implementations/) gets cheaper when rollout is
  invisible).
- **Autoscaling and spot capacity become usable aggressively** — scale-in is safe at any hour,
  and the spot discount stops costing correctness.
- **Handovers get fast**: explicit lease release and queue nacks mean successors take over in
  milliseconds instead of waiting out TTLs.

**Harder:**

- **The drain path is real code with real ordering constraints**, and it's exercised
  constantly — which is the good news wearing work clothes: bugs in it surface in daylight
  deploys, not 3 a.m. failovers.
- **Budgeting forces uncomfortable honesty** about your longest requests and slowest jobs; the
  request that can't finish inside any reasonable grace period was always a problem — shutdown
  design is merely where it stops being deniable.
- **Two disciplines, permanently**: the crash-safe substrate *and* the graceful layer, with the
  standing temptation to let the second quietly excuse gaps in the first. The `kill -9` habit
  is the immune system for exactly that drift.

## References

- [ZFN-37](/zfn/37-every-lock-is-a-lease/) — leases, expiry, and fencing: the machinery that
  makes both clean and unclean death safe for held work.
- [ZFN-19](/zfn/19-annotate-readonly-idempotent-endpoints/) — idempotency, which turns
  "interrupted and retried" into a non-event.
- [ZFN-13](/zfn/13-load-shedding-and-flow-control/) — draining as flow control, including the
  reconnect storm you create by draining carelessly.
- [ZFN-24](/zfn/24-one-transactional-store-per-write/) — durable facts live in the store, never
  in a buffer waiting on a shutdown hook.
- [Crash-Only Software](ref:crashonly) — the argument that stop=crash is the only shutdown
  contract you can trust; [ECS](ref:ecsshutdown) and [Kubernetes](ref:k8stermination) — the
  concrete choreography and the numbers to budget inside.

## Changelog

- **2026-08-12**: First published as a Field Note.