--- id: 64 title: "A config change is a deploy" kind: note status: current date: 2026-08-12 authors: - "Theo Zourzouvillys" tags: [config, reliability, operations, infra, deploy] references: - id: cf2019 title: "Details of the Cloudflare outage on July 2, 2019" url: https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/ abstract: "A single WAF rule — a regular expression with catastrophic backtracking — was pushed globally in seconds via a config pipeline that skipped the staged rollout code changes got, taking CPU to 100% on every machine in every city. The definitive postmortem of 'it was just config'." - id: sreconfig title: "Configuration Design and Best Practices (Google SRE Workbook)" url: https://sre.google/workbook/configuration-design/ abstract: "Google's distilled guidance on configuration as a system: config is code-adjacent, drives outages at comparable rates, and deserves versioning, validation, staged rollout, and the same review culture — with specific attention to blast radius and the danger of global, instant application." - id: fb2021 title: "More details about the October 4 outage (Meta Engineering)" url: https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/ abstract: "The six-hour global Facebook outage: a routine configuration command to backbone routers, an audit tool bug that failed to stop it, and a lockout that took down the tools needed for recovery — config change as total outage, including of the systems meant to fix it." summary: "Config changes cause outages as often as code — and ride to production with none of code's safeguards. Anything that changes production behaviour is a deploy, whatever file it lives in: versioned, validated, canaried, staged, observable, and revertible in one motion." supersedes: null superseded_by: null aliases: [] crossrefs: ZFN-16: "The consumption side of this note: data planes hold a validated last-known-good snapshot, so a bad config push degrades to 'no changes today' instead of 'no service today'." ZFN-17: "Config as one validated snapshot rather than a pile of live lookups is what makes 'canary a config version' a coherent sentence at all." ZFN-63: "Flag flips are the highest-frequency config deploys you run; the two notes describe one pipeline at two speeds." --- ## TL;DR **The question is never "is it code or config?" The question is "does it change production behaviour?" If yes, it's a deploy — and it gets a deploy's safeguards:** - **Versioned** in source control, with an author, a diff, and a review. - **Validated** before it ships — schema-checked, linted, and *evaluated against reality* (the [Cloudflare 2019 outage](ref:cf2019) was a regex whose pathological cost a pre-flight execution check would have caught). - **Staged** — canary instance, one cell, one region, the fleet — with health gates between steps. Global-and-instant is the outage shape, not a feature. - **Observable** — every instance reports which config version it's running, so "what changed?" is a query, not an investigation. - **Revertible in one motion**, to a known-good version that's still on hand. The corollary that stings: emergency changes ride the same pipeline. If the pipeline is too slow for an incident, **make the pipeline faster** — the side door you build instead will become the default path exactly when care matters most. ## Context Ask what took a system down and "a code bug" and "a config change" arrive at comparable rates — Google's SRE material treats [configuration-induced outage as a first-class category](ref:sreconfig). The reason isn't that config is written by careless people; it's structural. Code earns safeguards on its way to production: review, CI, canary, staged rollout, rollback machinery ([ZFN-63](/zfn/63-decouple-deploy-from-release/)). Config was born as "the safe part" — *just data*, tweaked by hand — and kept its innocence while accumulating power. A modern config file decides routing, quotas ([ZFN-18](/zfn/18-enforce-quotas-at-ingress/)), security policy, WAF rules, connection limits, which dependency to trust. It's the highest-privilege interpreter input in your fleet, and in many shops it still ships via an edit box and confidence. The two canonical postmortems bracket the failure space: - **[Cloudflare, 2019](ref:cf2019):** one WAF rule, pushed globally in seconds, through a config path that deliberately skipped the staged rollout code got (WAF rules need to ship fast — the reasoning sounds impeccable right up until it doesn't). CPU to 100%, every machine, every city. - **[Facebook, 2021](ref:fb2021):** a routine backbone-router config command, an audit check that should have stopped it and didn't, and six hours of global darkness — including the internal tools needed to fix it, which lived behind the network the config had just killed ([ZFN-4](/zfn/4-incident-tooling-independence/): the recovery tooling must not depend on what it recovers, and config blast radius is exactly how that dependency gets discovered). Both had world-class engineering cultures. Both got there via the same belief this note exists to kill: *the change was small, and it was only config.* Size-of-diff is a code heuristic; config's whole job is leverage — one line fans out to the entire fleet's behaviour. The blast radius of a config change is the set of things that read it, which is usually **everything, at once** — which is precisely why it deserves *more* rollout discipline than code, and traditionally gets none. > [!aside] > > The scariest phrase in an incident channel is "I'll just bump it in the console." Not > because the person is wrong about the value — they're usually right — but because the > change now exists nowhere: no diff, no reviewer, no version, invisible to the next > responder, silently reverted by the next real deploy, or worse, silently *not*. Console > drift is how a fleet ends up running a configuration that exists only as folklore. ## Recommendation **One pipeline for behaviour changes, whatever file format they wear.** - **Config lives in version control, full stop.** The running fleet's configuration is buildable from the repo at any commit; anything hand-touched in a console is drift, and drift detection pages someone. (Secrets are the one exception — the repo holds the *reference*, never the value: [ZFN-35](/zfn/35-dereference-secrets-not-store-in-config/).) - **Compile it, don't just parse it.** Validation has layers, and each has caught real outages: syntax; schema (types, ranges, cross-field invariants — [ZFN-17](/zfn/17-separate-config-state-ephemeral/)'s "one validated snapshot" is the natural unit); semantics (does this regex terminate in bounded time ([Cloudflare's lesson](ref:cf2019)), does this quota exceed downstream capacity, does this policy reference a principal that exists); and *simulation* where stakes justify it — evaluate the candidate config against a sample of live traffic and diff the decisions before any instance obeys it. - **Roll it out like code, gated on health.** Canary → cell → region → fleet, with the same dashboards a code deploy watches and automatic halt-and-revert on regression. This requires config to be **versioned as an artefact** (an immutable snapshot with an ID — [ZFN-17](/zfn/17-separate-config-state-ephemeral/)) rather than a bag of keys mutated in place; "instance X runs config version N" must be a fact the fleet can report ([ZFN-40](/zfn/40-no-anonymous-system-actor/) applies to config pushes too: every version has an author and a reason attached). - **Let consumers survive bad pushes.** Data planes hold last-known-good and keep serving when a new snapshot fails validation or fails to arrive ([ZFN-16](/zfn/16-separate-data-plane-control-plane/)); a config outage should degrade to "no changes land today," never "no traffic flows today." This is the property that turns a bad push from an outage into a rollback. - **Match the pipeline's speed to its fastest legitimate user, so nobody routes around it.** Flag flips ([ZFN-63](/zfn/63-decouple-deploy-from-release/)) and incident kill switches need seconds — fine: seconds through the pipeline, with validation, staging (even compressed), versioning, and the audit trail intact. The Cloudflare failure wasn't that WAF rules shipped fast; it's that shipping fast *exempted them from staging*. Speed and safeguards are orthogonal; the side door confuses them permanently — and the side door you keep "for emergencies" trains everyone to reach for it exactly when the fleet is least able to absorb a mistake. - **Rehearse the revert** ([ZFN-36](/zfn/36-test-backups-by-restoring/) energy): reverting to the previous config version is a one-command, regularly-exercised motion — because a revert nobody has run since the pipeline was built is a second incident wearing a rescue uniform. ## Consequences **Easier:** - **"What changed?" collapses to a query** — the first question of every incident gets an authoritative answer covering *all* behaviour changes, not just the code ones. - **Config regains trust.** Staged, health-gated pushes mean a bad value takes down a canary, not a fleet — and the postmortem is about a validation gap, not about who typed what. - **Audit and compliance come free** — author, review, version, and rollout history exist as a side effect of the pipeline rather than as a quarterly reconstruction. **Harder:** - **You've made changing a number bureaucratic**, and the pressure to exempt "trivial" changes will be constant, reasonable-sounding, and wrong — the trivial change with fleet fan-out is the exact shape of both canonical outages. The honest mitigation is pipeline speed, not exemptions. - **The machinery is real**: snapshot builds, version distribution, per-instance reporting, drift detection, simulation harnesses. It's a genuine platform investment ([ZFN-47](/zfn/47-govern-the-contract-between-teams/) — and worth sharing as one, not rebuilding per team). - **Some config genuinely resists staging** — global by nature (DNS, BGP, org-wide security policy) with no canary-shaped subset. Those changes keep manual ceremony: two-person rules, explicit checklists, and a pre-verified recovery path that doesn't depend on the thing being changed ([ZFN-4](/zfn/4-incident-tooling-independence/), [the Facebook lockout](ref:fb2021)). ## References - [ZFN-16](/zfn/16-separate-data-plane-control-plane/) — last-known-good consumption: the property that makes config pushes survivable at all. - [ZFN-17](/zfn/17-separate-config-state-ephemeral/) — config as a validated, versioned snapshot; the artefact this note's pipeline ships. - [ZFN-63](/zfn/63-decouple-deploy-from-release/) — flag flips as the high-frequency end of the same pipeline. - [ZFN-35](/zfn/35-dereference-secrets-not-store-in-config/) — the one thing config must reference rather than contain. - [ZFN-4](/zfn/4-incident-tooling-independence/) — why the recovery path must survive the config change that broke everything else. - [Cloudflare 2019](ref:cf2019) and [Facebook 2021](ref:fb2021) — the two bracketing postmortems; [SRE Workbook on configuration](ref:sreconfig) — the general case, stated by the people who run the most config. ## Changelog - **2026-08-12**: First published as a Field Note.