--- id: 63 title: "Decouple deploy from release — and give every flag a death date" kind: note status: current date: 2026-08-12 authors: - "Theo Zourzouvillys" tags: [process, reliability, operations, deploy, config] references: - id: toggles title: "Feature Toggles (aka Feature Flags) — Pete Hodgson, martinfowler.com" url: https://martinfowler.com/articles/feature-toggles.html abstract: "The reference taxonomy for flags: release toggles, experiment toggles, ops toggles, and permission toggles, each with a different natural lifetime and dynamism — and the argument that managing toggle inventory and retiring toggles aggressively is the difference between a technique and a mess." - id: knight title: "SEC Administrative Proceeding — In the Matter of Knight Capital Americas LLC (2013)" url: https://www.sec.gov/litigation/admin/2013/34-70694.pdf abstract: "The regulator's account of the canonical dead-flag disaster: Knight repurposed a flag that still gated eight-year-old dead code, a deploy missed one of eight servers, and the flag activated the old path — 4 million unintended orders and a $460 million loss in 45 minutes." summary: "A deploy puts code on servers; a release changes what users see. Coupled, a deploy is a bet you can only unwind by redeploying. Decoupled by flags, deploys become boring and releases progressive and instantly reversible. But a flag is a loan: owner, death date, or Knight Capital." supersedes: null superseded_by: null aliases: [] crossrefs: ZFN-62: "The database-side twin: expand/migrate/contract is deploy/release decoupling for schemas, and the flag that moves readers is its release lever." ZFN-64: "A flag flip is a production change and deserves a deploy's safeguards — the two notes are one position stated from opposite ends." ZFN-16: "Flag evaluation is a control-plane read on the data plane's hot path — local snapshot, last-known-good, never a synchronous fetch per request." --- ## TL;DR **A deploy is code arriving on servers. A release is users experiencing new behaviour. Fuse them and you've made every rollout a high-stakes bet whose only undo is another deploy — so deploys become rare, big, and frightening, which makes them riskier still. Split them:** - **Deploy continuously and invisibly** — new code ships dark, weeks of small deploys, nothing changes for users ([ZFN-62](/zfn/62-expand-migrate-contract/) runs on exactly this). - **Release progressively and reversibly** — a flag flip, rolled out by percentage, cohort, or tenant, watched against metrics, and revertible in **seconds without a deploy**. The kill switch for a bad release should be the cheapest control you own. - **But treat every flag as a loan against your codebase.** Each one doubles a state space nobody fully tests. So: **an owner and a death date at creation**, removal as part of the release's definition of done, and an inventory someone actually reads ([Hodgson's taxonomy](ref:toggles) — release, experiment, ops, permission — because the four kinds have four different lifetimes, and confusing them is how flags become permanent). - **Never repurpose a flag, and never leave dead code behind one.** [Knight Capital](ref:knight) is the forty-five-minute, $460M proof of both rules at once. ## Context When deploy and release are the same event, the incentives compound badly. Shipping anything user-visible means shipping *everything* since the last release, so releases batch up; batched releases are risky, so they get ceremonies — freeze windows, release managers, Thursday-only — which slows the loop further; and when one goes wrong, the only remedy is a rollback that takes *everything else* down with it, so teams press forward through incidents instead. Every pathology feeds the next. Decoupled, each half gets to be good at its own job. Deploys optimise for **small and frequent** — tiny diffs, trivially bisectable, [rollback always cheap](/zfn/62-expand-migrate-contract/) because nothing user-facing depended on the timing. Releases optimise for **controlled exposure** — 1% of traffic, then one tenant cohort, then everyone, each step gated on the dashboards, with retreat costing one flag flip. The blast radius of "we were wrong" drops from "everyone, until we can redeploy" to "1% of traffic, for ninety seconds." Then the bill arrives, because the mechanism that buys this is *conditional behaviour in production code*, and it accumulates: - **Every live flag doubles the theoretical state space.** Ten interacting flags is a thousand configurations; CI tests two or three. The untested combinations don't stay hypothetical — they're what's actually running for whichever cohort has the unlucky mix. - **Stale flags rot into landmines.** The flag everyone believes is permanently-on still has an off-path — untested for two years, wired to who-knows-what — and one config mistake or emergency "just flip something" away from executing. [Knight Capital](ref:knight) is the named version: a flag repurposed for new behaviour while eight-year-old dead code still listened to it on one path; a deploy that missed one server did the rest. - **Flag checks metastasise.** The same flag consulted in five services renders five independent opinions during a rollout — the mixed state *is* the bug ([ZFN-51](/zfn/51-design-the-request-envelope-first/): if a decision must be consistent across a request, decide once at the edge and propagate the decision, not the question). > [!aside] > > A useful reframe for review: a feature flag is an `if` statement with production write > access and no test coverage on one branch. You would never merge that sentence as code > review feedback; the flag platform just makes it look tidy. ## Recommendation **Decouple fully, then govern the mechanism like the liability it is.** - **Dark-ship by default.** Merges deploy behind an off flag as a matter of course; "deployed" and "released" become separately-answerable questions in every incident and every standup. This is also what makes trunk-based, small-batch development safe at all ([ZFN-23](/zfn/23-iterate-and-rewrite-implementations/)'s rewrites ride the same trick: new implementation dark alongside old, cut over by flag, old path deleted after). - **Release as a ramp with hands on the numbers.** Percentage or cohort rollout, error and latency dashboards named *in the release plan*, automatic or one-click retreat. Tenant-aware products ramp by tenant ([ZFN-15](/zfn/15-partition-customer-data-by-tenant/)) — your riskiest cohort shouldn't be in the first percent. - **Evaluate flags locally, off a pushed snapshot.** Flag state is control-plane data; the data plane consumes a validated local copy with last-known-good semantics, never a synchronous per-request fetch ([ZFN-16](/zfn/16-separate-data-plane-control-plane/), [ZFN-17](/zfn/17-separate-config-state-ephemeral/)). The flag service being down must mean "no releases today," never "no traffic today." - **Separate the four kinds, mechanically** ([the taxonomy](ref:toggles)): **release flags** (lifespan: one rollout — the ones this note is mostly about), **experiment flags** (lifespan: the experiment, owned by its analysis), **ops/kill switches** (long-lived *on purpose*, few, inventoried, tested regularly — these are [incident tooling](/zfn/4-incident-tooling-independence/) and earn their permanence), and **entitlements** (not flags at all — product configuration, living in the customer model, not the flag system). Most flag-system rot is one of the other three kinds squatting in the release-flag lane. - **Create every release flag with its death written down**: an owner, an intended removal milestone, and a tracker entry filed at creation. The release isn't *done* at 100% — it's done when the flag and the old code path are **deleted**. Aging flags page their owner, not a dashboard nobody opens. And two absolute rules, both Knight-shaped: dead code is removed *before* its flag is, and a flag name is never reused for new semantics. - **Test the states that can actually occur**: both sides of every live release flag in CI, plus the specific combinations the ramp will pass through. Combinatorial coverage is impossible; *scheduled-state* coverage isn't, and it's the difference between a rollout plan and a hope. ## Consequences **Easier:** - **Deploys stop being events** — small, constant, boring, and rollback-safe, which is the soil everything else here grows in ([ZFN-62](/zfn/62-expand-migrate-contract/), [ZFN-60](/zfn/60-graceful-shutdown-is-a-protocol/)). - **Bad releases cost seconds, not incident-lengths** — and the retreat doesn't take unrelated work down with it. - **Release timing becomes a product decision** — coordinated launches, tenant-by-tenant enterprise rollouts, marketing-aligned flips — with zero engineering ceremony attached. **Harder:** - **You now run a flag platform**, and it's on the hot path of everything: its snapshot distribution, its evaluation consistency, and its audit trail ([ZFN-64](/zfn/64-config-changes-are-deploys/): a flip is a production change and gets a production change's log) are real infrastructure with real failure modes. - **The inventory discipline is unglamorous and permanent** — flag review is weeding, forever; skip a few quarters and you're back to archaeology, except now it's archaeology that can be toggled. - **Code with both paths in it is genuinely harder to read**, and the discipline of writing the new path as if the old one were already gone — then deleting on schedule — is a taste that has to be taught in review. ## References - [ZFN-62](/zfn/62-expand-migrate-contract/) — the schema-side twin; its migrate phase is a release flag doing its one job and then dying on time. - [ZFN-64](/zfn/64-config-changes-are-deploys/) — the flip itself is a config deploy: validated, staged, observable, revertible. - [ZFN-16](/zfn/16-separate-data-plane-control-plane/) — why flag reads are local snapshots with last-known-good, never live lookups. - [ZFN-4](/zfn/4-incident-tooling-independence/) — kill switches as incident tooling: the one flag species that earns permanence, and the standards it must meet to keep it. - [Feature Toggles](ref:toggles) — the taxonomy and the retirement discipline; [the Knight Capital order](ref:knight) — what a repurposed flag over dead code costs, with the regulator doing the counting. ## Changelog - **2026-08-12**: First published as a Field Note.