A config change is a deploy

Context

Ask what took a system down and “a code bug” and “a config change” arrive at comparable rates — Google’s SRE material treats configuration-induced outage as a first-class categoryConfiguration Design and Best Practices (Google SRE Workbook)Google's distilled guidance on configuration as a system: config is code-adjacent, drives outages at comparable rates, and deserves versioning, validation, staged rollout, and the same review culture — with specific attention to blast radius and the danger of global, instant application.sre.google ↗. The reason isn’t that config is written by careless people; it’s structural. Code earns safeguards on its way to production: review, CI, canary, staged rollout, rollback machinery (ZFN-63Field Note · currentZFN-63 — Decouple deploy from release — and give every flag a death dateA deploy puts code on servers; a release changes what users see. Coupled, a deploy is a bet you can only unwind by redeploying. Decoupled by flags, deploys become boring and releases progressive and instantly reversible. But a flag is a loan: owner, death date, or Knight Capital.Why it's cited here: Flag flips are the highest-frequency config deploys you run; the two notes describe one pipeline at two speeds.Open ZFN-63 →). Config was born as “the safe part” — just data, tweaked by hand — and kept its innocence while accumulating power. A modern config file decides routing, quotas (ZFN-18Field Note · currentZFN-18 — Enforce a quota at ingress on every endpoint — even unabused onesPut a quota on every endpoint and enforce it at ingress from day one — per tenant, principal, IP — even for endpoints nobody abuses yet. Unlimited-by-default means the first runaway client or compromised key is an outage. Return 429 + Retry-After; retrofitting limits is painful.Open ZFN-18 →), security policy, WAF rules, connection limits, which dependency to trust. It’s the highest-privilege interpreter input in your fleet, and in many shops it still ships via an edit box and confidence.

The two canonical postmortems bracket the failure space:

• Cloudflare, 2019Details of the Cloudflare outage on July 2, 2019A single WAF rule — a regular expression with catastrophic backtracking — was pushed globally in seconds via a config pipeline that skipped the staged rollout code changes got, taking CPU to 100% on every machine in every city. The definitive postmortem of 'it was just config'.blog.cloudflare.com ↗: one WAF rule, pushed globally in seconds, through a config path that deliberately skipped the staged rollout code got (WAF rules need to ship fast — the reasoning sounds impeccable right up until it doesn’t). CPU to 100%, every machine, every city.
• Facebook, 2021More details about the October 4 outage (Meta Engineering)The six-hour global Facebook outage: a routine configuration command to backbone routers, an audit tool bug that failed to stop it, and a lockout that took down the tools needed for recovery — config change as total outage, including of the systems meant to fix it.engineering.fb.com ↗: a routine backbone-router config command, an audit check that should have stopped it and didn’t, and six hours of global darkness — including the internal tools needed to fix it, which lived behind the network the config had just killed (ZFN-4Field Note · currentZFN-4 — Incident tooling must not depend on what it recoversAnything you need to respond to an incident — deploy/rollback, kill switches, observability, break-glass access — must not depend, directly or transitively, on the systems likely to be down during it. Never gate incident tooling behind a system it might need to recover.Open ZFN-4 →: the recovery tooling must not depend on what it recovers, and config blast radius is exactly how that dependency gets discovered).

Both had world-class engineering cultures. Both got there via the same belief this note exists to kill: the change was small, and it was only config. Size-of-diff is a code heuristic; config’s whole job is leverage — one line fans out to the entire fleet’s behaviour. The blast radius of a config change is the set of things that read it, which is usually everything, at once — which is precisely why it deserves more rollout discipline than code, and traditionally gets none.

Recommendation

One pipeline for behaviour changes, whatever file format they wear.

• Config lives in version control, full stop. The running fleet’s configuration is buildable from the repo at any commit; anything hand-touched in a console is drift, and drift detection pages someone. (Secrets are the one exception — the repo holds the reference, never the value: ZFN-35Field Note · currentZFN-35 — Reference secrets in config; dereference, refresh, and re-fetchDon't put secret values in config — store a reference (a path in a secret store) and dereference it at runtime via your workload identity. Refresh on a signal or expiry so rotation needs no redeploy; re-fetch on auth failure so a rotated secret self-heals.Open ZFN-35 →.)

• Compile it, don’t just parse it. Validation has layers, and each has caught real outages: syntax; schema (types, ranges, cross-field invariants — ZFN-17Field Note · currentZFN-17 — Separate configuration, state, and ephemeral dataCustomer data splits into mostly-static config, durable state, and ephemeral sessions — different access, durability, and change rates. Model and store each separately. For bounded static config, prefer loading one validated snapshot held in memory over fetching on demand.Why it's cited here: Config as one validated snapshot rather than a pile of live lookups is what makes 'canary a config version' a coherent sentence at all.Open ZFN-17 →’s “one validated snapshot” is the natural unit); semantics (does this regex terminate in bounded time (Cloudflare’s lessonDetails of the Cloudflare outage on July 2, 2019A single WAF rule — a regular expression with catastrophic backtracking — was pushed globally in seconds via a config pipeline that skipped the staged rollout code changes got, taking CPU to 100% on every machine in every city. The definitive postmortem of 'it was just config'.blog.cloudflare.com ↗), does this quota exceed downstream capacity, does this policy reference a principal that exists); and simulation where stakes justify it — evaluate the candidate config against a sample of live traffic and diff the decisions before any instance obeys it.

• Roll it out like code, gated on health. Canary → cell → region → fleet, with the same dashboards a code deploy watches and automatic halt-and-revert on regression. This requires config to be versioned as an artefact (an immutable snapshot with an ID — ZFN-17Field Note · currentZFN-17 — Separate configuration, state, and ephemeral dataCustomer data splits into mostly-static config, durable state, and ephemeral sessions — different access, durability, and change rates. Model and store each separately. For bounded static config, prefer loading one validated snapshot held in memory over fetching on demand.Why it's cited here: Config as one validated snapshot rather than a pile of live lookups is what makes 'canary a config version' a coherent sentence at all.Open ZFN-17 →) rather than a bag of keys mutated in place; “instance X runs config version N” must be a fact the fleet can report (ZFN-40Field Note · currentZFN-40 — No anonymous "system" actorIf "system" appears as an actor in your audit log, attribution is already broken. Every automated action — cron job, cleanup task, migration, agent — runs as a named identity with its own credentials and scope, so "who did this?" has an answer and revocation is surgical.Open ZFN-40 → applies to config pushes too: every version has an author and a reason attached).

• Let consumers survive bad pushes. Data planes hold last-known-good and keep serving when a new snapshot fails validation or fails to arrive (ZFN-16Field Note · currentZFN-16 — Separate the data plane from the control planeSplit the serving path (data plane) from the management path (control plane). The data plane keeps serving on last-known-good config when the control plane is down — never call it on the hot path. Coupling them turns a control-plane bug into a serving outage.Why it's cited here: The consumption side of this note: data planes hold a validated last-known-good snapshot, so a bad config push degrades to 'no changes today' instead of 'no service today'.Open ZFN-16 →); a config outage should degrade to “no changes land today,” never “no traffic flows today.” This is the property that turns a bad push from an outage into a rollback.

• Match the pipeline’s speed to its fastest legitimate user, so nobody routes around it. Flag flips (ZFN-63Field Note · currentZFN-63 — Decouple deploy from release — and give every flag a death dateA deploy puts code on servers; a release changes what users see. Coupled, a deploy is a bet you can only unwind by redeploying. Decoupled by flags, deploys become boring and releases progressive and instantly reversible. But a flag is a loan: owner, death date, or Knight Capital.Why it's cited here: Flag flips are the highest-frequency config deploys you run; the two notes describe one pipeline at two speeds.Open ZFN-63 →) and incident kill switches need seconds — fine: seconds through the pipeline, with validation, staging (even compressed), versioning, and the audit trail intact. The Cloudflare failure wasn’t that WAF rules shipped fast; it’s that shipping fast exempted them from staging. Speed and safeguards are orthogonal; the side door confuses them permanently — and the side door you keep “for emergencies” trains everyone to reach for it exactly when the fleet is least able to absorb a mistake.

• Rehearse the revert (ZFN-36Field Note · currentZFN-36 — An untested backup is not a backup — test it by restoringAn untested backup is a hope, not a backup — the only thing that counts is a restore. Rehearse restores regularly (game days), measure and meet your RTO/RPO, automate them, and cover the whole recovery path — data, schema, config, secrets, cutover — not just the dump.Open ZFN-36 → energy): reverting to the previous config version is a one-command, regularly-exercised motion — because a revert nobody has run since the pipeline was built is a second incident wearing a rescue uniform.

Consequences

Easier:

• “What changed?” collapses to a query — the first question of every incident gets an authoritative answer covering all behaviour changes, not just the code ones.
• Config regains trust. Staged, health-gated pushes mean a bad value takes down a canary, not a fleet — and the postmortem is about a validation gap, not about who typed what.
• Audit and compliance come free — author, review, version, and rollout history exist as a side effect of the pipeline rather than as a quarterly reconstruction.

Harder:

• You’ve made changing a number bureaucratic, and the pressure to exempt “trivial” changes will be constant, reasonable-sounding, and wrong — the trivial change with fleet fan-out is the exact shape of both canonical outages. The honest mitigation is pipeline speed, not exemptions.
• The machinery is real: snapshot builds, version distribution, per-instance reporting, drift detection, simulation harnesses. It’s a genuine platform investment (ZFN-47Field Note · currentZFN-47 — Govern the contract between teams, not the code inside themTeams own services end to end; one team owns the gateway that dispatches to them. Govern exactly one thing centrally — the contract at the boundary (schema, identity, errors, idempotency) — and enforce it at runtime. Don't mandate libraries; ship them as an opt-in blueprint.Open ZFN-47 → — and worth sharing as one, not rebuilding per team).
• Some config genuinely resists staging — global by nature (DNS, BGP, org-wide security policy) with no canary-shaped subset. Those changes keep manual ceremony: two-person rules, explicit checklists, and a pre-verified recovery path that doesn’t depend on the thing being changed (ZFN-4Field Note · currentZFN-4 — Incident tooling must not depend on what it recoversAnything you need to respond to an incident — deploy/rollback, kill switches, observability, break-glass access — must not depend, directly or transitively, on the systems likely to be down during it. Never gate incident tooling behind a system it might need to recover.Open ZFN-4 →, the Facebook lockoutMore details about the October 4 outage (Meta Engineering)The six-hour global Facebook outage: a routine configuration command to backbone routers, an audit tool bug that failed to stop it, and a lockout that took down the tools needed for recovery — config change as total outage, including of the systems meant to fix it.engineering.fb.com ↗).

References

• ZFN-16Field Note · currentZFN-16 — Separate the data plane from the control planeSplit the serving path (data plane) from the management path (control plane). The data plane keeps serving on last-known-good config when the control plane is down — never call it on the hot path. Coupling them turns a control-plane bug into a serving outage.Why it's cited here: The consumption side of this note: data planes hold a validated last-known-good snapshot, so a bad config push degrades to 'no changes today' instead of 'no service today'.Open ZFN-16 → — last-known-good consumption: the property that makes config pushes survivable at all.
• ZFN-17Field Note · currentZFN-17 — Separate configuration, state, and ephemeral dataCustomer data splits into mostly-static config, durable state, and ephemeral sessions — different access, durability, and change rates. Model and store each separately. For bounded static config, prefer loading one validated snapshot held in memory over fetching on demand.Why it's cited here: Config as one validated snapshot rather than a pile of live lookups is what makes 'canary a config version' a coherent sentence at all.Open ZFN-17 → — config as a validated, versioned snapshot; the artefact this note’s pipeline ships.
• ZFN-63Field Note · currentZFN-63 — Decouple deploy from release — and give every flag a death dateA deploy puts code on servers; a release changes what users see. Coupled, a deploy is a bet you can only unwind by redeploying. Decoupled by flags, deploys become boring and releases progressive and instantly reversible. But a flag is a loan: owner, death date, or Knight Capital.Why it's cited here: Flag flips are the highest-frequency config deploys you run; the two notes describe one pipeline at two speeds.Open ZFN-63 → — flag flips as the high-frequency end of the same pipeline.
• ZFN-35Field Note · currentZFN-35 — Reference secrets in config; dereference, refresh, and re-fetchDon't put secret values in config — store a reference (a path in a secret store) and dereference it at runtime via your workload identity. Refresh on a signal or expiry so rotation needs no redeploy; re-fetch on auth failure so a rotated secret self-heals.Open ZFN-35 → — the one thing config must reference rather than contain.
• ZFN-4Field Note · currentZFN-4 — Incident tooling must not depend on what it recoversAnything you need to respond to an incident — deploy/rollback, kill switches, observability, break-glass access — must not depend, directly or transitively, on the systems likely to be down during it. Never gate incident tooling behind a system it might need to recover.Open ZFN-4 → — why the recovery path must survive the config change that broke everything else.
• Cloudflare 2019Details of the Cloudflare outage on July 2, 2019A single WAF rule — a regular expression with catastrophic backtracking — was pushed globally in seconds via a config pipeline that skipped the staged rollout code changes got, taking CPU to 100% on every machine in every city. The definitive postmortem of 'it was just config'.blog.cloudflare.com ↗ and Facebook 2021More details about the October 4 outage (Meta Engineering)The six-hour global Facebook outage: a routine configuration command to backbone routers, an audit tool bug that failed to stop it, and a lockout that took down the tools needed for recovery — config change as total outage, including of the systems meant to fix it.engineering.fb.com ↗ — the two bracketing postmortems; SRE Workbook on configurationConfiguration Design and Best Practices (Google SRE Workbook)Google's distilled guidance on configuration as a system: config is code-adjacent, drives outages at comparable rates, and deserves versioning, validation, staged rollout, and the same review culture — with specific attention to blast radius and the danger of global, instant application.sre.google ↗ — the general case, stated by the people who run the most config.

Changelog

• 2026-08-12: First published as a Field Note.