Decouple deploy from release — and give every flag a death date

Context

When deploy and release are the same event, the incentives compound badly. Shipping anything user-visible means shipping everything since the last release, so releases batch up; batched releases are risky, so they get ceremonies — freeze windows, release managers, Thursday-only — which slows the loop further; and when one goes wrong, the only remedy is a rollback that takes everything else down with it, so teams press forward through incidents instead. Every pathology feeds the next.

Decoupled, each half gets to be good at its own job. Deploys optimise for small and frequent — tiny diffs, trivially bisectable, rollback always cheapField Note · currentZFN-62 — Expand, migrate, contract: schema changes in three movesEvery deploy runs two code versions against one database — and rollback runs yesterday's code on today's schema. No schema change may break either. So every migration is three shippable moves: expand (additive), migrate (backfill, verify), contract (remove, later, deliberately).Why it's cited here: The database-side twin: expand/migrate/contract is deploy/release decoupling for schemas, and the flag that moves readers is its release lever.Open ZFN-62 → because nothing user-facing depended on the timing. Releases optimise for controlled exposure — 1% of traffic, then one tenant cohort, then everyone, each step gated on the dashboards, with retreat costing one flag flip. The blast radius of “we were wrong” drops from “everyone, until we can redeploy” to “1% of traffic, for ninety seconds.”

Then the bill arrives, because the mechanism that buys this is conditional behaviour in production code, and it accumulates:

• Every live flag doubles the theoretical state space. Ten interacting flags is a thousand configurations; CI tests two or three. The untested combinations don’t stay hypothetical — they’re what’s actually running for whichever cohort has the unlucky mix.
• Stale flags rot into landmines. The flag everyone believes is permanently-on still has an off-path — untested for two years, wired to who-knows-what — and one config mistake or emergency “just flip something” away from executing. Knight CapitalSEC Administrative Proceeding — In the Matter of Knight Capital Americas LLC (2013)The regulator's account of the canonical dead-flag disaster: Knight repurposed a flag that still gated eight-year-old dead code, a deploy missed one of eight servers, and the flag activated the old path — 4 million unintended orders and a $460 million loss in 45 minutes.sec.gov ↗ is the named version: a flag repurposed for new behaviour while eight-year-old dead code still listened to it on one path; a deploy that missed one server did the rest.
• Flag checks metastasise. The same flag consulted in five services renders five independent opinions during a rollout — the mixed state is the bug (ZFN-51Field Note · currentZFN-51 — Design the request envelope before the first endpointAuth, idempotency keys, trace IDs, vector clocks: context about a request, not part of it. Define an envelope alongside the payload in the schema on day one — transport-native, per-hop vs propagated, owned by generated SDKs and middleware. The retrofit is what costs you.Open ZFN-51 →: if a decision must be consistent across a request, decide once at the edge and propagate the decision, not the question).

Recommendation

Decouple fully, then govern the mechanism like the liability it is.

• Dark-ship by default. Merges deploy behind an off flag as a matter of course; “deployed” and “released” become separately-answerable questions in every incident and every standup. This is also what makes trunk-based, small-batch development safe at all (ZFN-23Field Note · currentZFN-23 — Rewriting an implementation is fine — refactoring isn't always the answerRefactoring isn't always right. When the structure is wrong at the root, it's fine — often better — to rewrite an implementation from scratch. Clean interfaces and data models make the implementation disposable: stable contract, swappable internals. LLMs make it cheaper still.Open ZFN-23 →’s rewrites ride the same trick: new implementation dark alongside old, cut over by flag, old path deleted after).

• Release as a ramp with hands on the numbers. Percentage or cohort rollout, error and latency dashboards named in the release plan, automatic or one-click retreat. Tenant-aware products ramp by tenant (ZFN-15Field Note · currentZFN-15 — Partition customer data by tenant from day oneMake customer data tenant-partitioned from day one: tenant-scope every query, never join across tenants, route through a tenant→location directory. Run one physical database at first — but keep the model shardable. Retrofitting isolation onto a shared DB is brutal.Open ZFN-15 →) — your riskiest cohort shouldn’t be in the first percent.

• Evaluate flags locally, off a pushed snapshot. Flag state is control-plane data; the data plane consumes a validated local copy with last-known-good semantics, never a synchronous per-request fetch (ZFN-16Field Note · currentZFN-16 — Separate the data plane from the control planeSplit the serving path (data plane) from the management path (control plane). The data plane keeps serving on last-known-good config when the control plane is down — never call it on the hot path. Coupling them turns a control-plane bug into a serving outage.Why it's cited here: Flag evaluation is a control-plane read on the data plane's hot path — local snapshot, last-known-good, never a synchronous fetch per request.Open ZFN-16 →, ZFN-17Field Note · currentZFN-17 — Separate configuration, state, and ephemeral dataCustomer data splits into mostly-static config, durable state, and ephemeral sessions — different access, durability, and change rates. Model and store each separately. For bounded static config, prefer loading one validated snapshot held in memory over fetching on demand.Open ZFN-17 →). The flag service being down must mean “no releases today,” never “no traffic today.”

• Separate the four kinds, mechanically (the taxonomyFeature Toggles (aka Feature Flags) — Pete Hodgson, martinfowler.comThe reference taxonomy for flags: release toggles, experiment toggles, ops toggles, and permission toggles, each with a different natural lifetime and dynamism — and the argument that managing toggle inventory and retiring toggles aggressively is the difference between a technique and a mess.martinfowler.com ↗): release flags (lifespan: one rollout — the ones this note is mostly about), experiment flags (lifespan: the experiment, owned by its analysis), ops/kill switches (long-lived on purpose, few, inventoried, tested regularly — these are incident toolingField Note · currentZFN-4 — Incident tooling must not depend on what it recoversAnything you need to respond to an incident — deploy/rollback, kill switches, observability, break-glass access — must not depend, directly or transitively, on the systems likely to be down during it. Never gate incident tooling behind a system it might need to recover.Open ZFN-4 → and earn their permanence), and entitlements (not flags at all — product configuration, living in the customer model, not the flag system). Most flag-system rot is one of the other three kinds squatting in the release-flag lane.

• Create every release flag with its death written down: an owner, an intended removal milestone, and a tracker entry filed at creation. The release isn’t done at 100% — it’s done when the flag and the old code path are deleted. Aging flags page their owner, not a dashboard nobody opens. And two absolute rules, both Knight-shaped: dead code is removed before its flag is, and a flag name is never reused for new semantics.

• Test the states that can actually occur: both sides of every live release flag in CI, plus the specific combinations the ramp will pass through. Combinatorial coverage is impossible; scheduled-state coverage isn’t, and it’s the difference between a rollout plan and a hope.

Consequences

Easier:

• Deploys stop being events — small, constant, boring, and rollback-safe, which is the soil everything else here grows in (ZFN-62Field Note · currentZFN-62 — Expand, migrate, contract: schema changes in three movesEvery deploy runs two code versions against one database — and rollback runs yesterday's code on today's schema. No schema change may break either. So every migration is three shippable moves: expand (additive), migrate (backfill, verify), contract (remove, later, deliberately).Why it's cited here: The database-side twin: expand/migrate/contract is deploy/release decoupling for schemas, and the flag that moves readers is its release lever.Open ZFN-62 →, ZFN-60Field Note · currentZFN-60 — Drain before you die: graceful shutdown is a protocolSIGTERM isn't an emergency — it's every deploy and scale-in. Shutdown is a protocol: stop attracting work, drain while the balancer catches up, hand back in-flight work, release leases, exit before SIGKILL. But graceful is only the optimisation — crash-safe is the requirement.Open ZFN-60 →).
• Bad releases cost seconds, not incident-lengths — and the retreat doesn’t take unrelated work down with it.
• Release timing becomes a product decision — coordinated launches, tenant-by-tenant enterprise rollouts, marketing-aligned flips — with zero engineering ceremony attached.

Harder:

• You now run a flag platform, and it’s on the hot path of everything: its snapshot distribution, its evaluation consistency, and its audit trail (ZFN-64Field Note · currentZFN-64 — A config change is a deployConfig changes cause outages as often as code — and ride to production with none of code's safeguards. Anything that changes production behaviour is a deploy, whatever file it lives in: versioned, validated, canaried, staged, observable, and revertible in one motion.Why it's cited here: A flag flip is a production change and deserves a deploy's safeguards — the two notes are one position stated from opposite ends.Open ZFN-64 →: a flip is a production change and gets a production change’s log) are real infrastructure with real failure modes.
• The inventory discipline is unglamorous and permanent — flag review is weeding, forever; skip a few quarters and you’re back to archaeology, except now it’s archaeology that can be toggled.
• Code with both paths in it is genuinely harder to read, and the discipline of writing the new path as if the old one were already gone — then deleting on schedule — is a taste that has to be taught in review.

References

• ZFN-62Field Note · currentZFN-62 — Expand, migrate, contract: schema changes in three movesEvery deploy runs two code versions against one database — and rollback runs yesterday's code on today's schema. No schema change may break either. So every migration is three shippable moves: expand (additive), migrate (backfill, verify), contract (remove, later, deliberately).Why it's cited here: The database-side twin: expand/migrate/contract is deploy/release decoupling for schemas, and the flag that moves readers is its release lever.Open ZFN-62 → — the schema-side twin; its migrate phase is a release flag doing its one job and then dying on time.
• ZFN-64Field Note · currentZFN-64 — A config change is a deployConfig changes cause outages as often as code — and ride to production with none of code's safeguards. Anything that changes production behaviour is a deploy, whatever file it lives in: versioned, validated, canaried, staged, observable, and revertible in one motion.Why it's cited here: A flag flip is a production change and deserves a deploy's safeguards — the two notes are one position stated from opposite ends.Open ZFN-64 → — the flip itself is a config deploy: validated, staged, observable, revertible.
• ZFN-16Field Note · currentZFN-16 — Separate the data plane from the control planeSplit the serving path (data plane) from the management path (control plane). The data plane keeps serving on last-known-good config when the control plane is down — never call it on the hot path. Coupling them turns a control-plane bug into a serving outage.Why it's cited here: Flag evaluation is a control-plane read on the data plane's hot path — local snapshot, last-known-good, never a synchronous fetch per request.Open ZFN-16 → — why flag reads are local snapshots with last-known-good, never live lookups.
• ZFN-4Field Note · currentZFN-4 — Incident tooling must not depend on what it recoversAnything you need to respond to an incident — deploy/rollback, kill switches, observability, break-glass access — must not depend, directly or transitively, on the systems likely to be down during it. Never gate incident tooling behind a system it might need to recover.Open ZFN-4 → — kill switches as incident tooling: the one flag species that earns permanence, and the standards it must meet to keep it.
• Feature TogglesFeature Toggles (aka Feature Flags) — Pete Hodgson, martinfowler.comThe reference taxonomy for flags: release toggles, experiment toggles, ops toggles, and permission toggles, each with a different natural lifetime and dynamism — and the argument that managing toggle inventory and retiring toggles aggressively is the difference between a technique and a mess.martinfowler.com ↗ — the taxonomy and the retirement discipline; the Knight Capital orderSEC Administrative Proceeding — In the Matter of Knight Capital Americas LLC (2013)The regulator's account of the canonical dead-flag disaster: Knight repurposed a flag that still gated eight-year-old dead code, a deploy missed one of eight servers, and the flag activated the old path — 4 million unintended orders and a $460 million loss in 45 minutes.sec.gov ↗ — what a repurposed flag over dead code costs, with the regulator doing the counting.

Changelog

• 2026-08-12: First published as a Field Note.