Expand, migrate, contract: schema changes in three moves

Context

The deploy-time overlap isn’t an edge case you can squeeze out: rolling deploys, canaries, draining instances still serving during shutdownField Note · currentZFN-60 — Drain before you die: graceful shutdown is a protocolSIGTERM isn't an emergency — it's every deploy and scale-in. Shutdown is a protocol: stop attracting work, drain while the balancer catches up, hand back in-flight work, release leases, exit before SIGKILL. But graceful is only the optimisation — crash-safe is the requirement.Open ZFN-60 → — heterogeneous code against one schema is the normal operating condition of a continuously deployed system. And the rollback case is stricter and always forgotten: rollback is your primary incident remedy, it must stay cheap (ZFN-63Field Note · currentZFN-63 — Decouple deploy from release — and give every flag a death dateA deploy puts code on servers; a release changes what users see. Coupled, a deploy is a bet you can only unwind by redeploying. Decoupled by flags, deploys become boring and releases progressive and instantly reversible. But a flag is a loan: owner, death date, or Knight Capital.Why it's cited here: Deploy/release decoupling is the code-side twin: the flag that flips reads to the new column is a release decision, reversible in seconds, independent of the deploy that shipped it.Open ZFN-63 →), and it means last week’s code meeting this week’s schema. A schema change that breaks version N−1 hasn’t just constrained the deploy — it has quietly disabled the undo button.

What goes wrong without the discipline:

• The atomic rename. ALTER TABLE … RENAME COLUMN plus the code change in one release: every not-yet-updated instance breaks the moment the DDL commits, and rollback breaks the updated ones. The outage window is the deploy window, and the only exit is forward.
• The innocent-looking lock. DDL that looks instant can take a table lock and rewrite or scan the whole thing — behind it, every query queues, the connection pool fills, and the app is down without a single error in the migration log. Engines differ on which operations bite (and versions keep improving it); the rule that doesn’t change is: know the locking behaviour of every DDL statement you run against a hot table, in your engine, at your size — and where in-place is unacceptable, use the shadow-table machinery (gh-ostgh-ost — GitHub's online schema migration tool for MySQLA triggerless online schema-change tool: builds a shadow table, replays changes from the binlog, and cuts over when caught up. Exists because in-place DDL on large hot tables is operationally unacceptable — the tooling embodiment of 'never block writes for a schema change'.github.com ↗ and friends) built for exactly this.
• The app-boot migration. Migrations run by whichever instance boots first couple schema change to deploy topology: N instances race, a slow migration makes deploy timeouts kill half-applied work, and rollback re-runs nothing. Migration execution is its own pipeline step with its own logs, locks, and human-visible outcome.

The through-line: a database schema is an API with the strictest compatibility contract you operate — its consumers can’t be simultaneously updated, ever. You already hold APIs to one-version compatibility with CI-enforced checks (ZFN-14Field Note · currentZFN-14 — Define every API with a schema, and generate the clientsDefine every API with a machine-readable schema (OpenAPI, Protobuf, GraphQL) as the source of truth, and generate clients and server stubs from it — never hand-roll request-building and JSON parsing. Hand-written clients drift and break silently; check schema compatibility in CI.Why it's cited here: The same compatibility law one layer up: schema-first APIs get compatibility checked in CI; the database deserves the identical gate, because rollback is a consumer too.Open ZFN-14 →); the schema deserves the same law, with rollback counted as a consumer.

Recommendation

Run every schema change as three reviewed, independently-deployed moves — with evidence gates between them.

• Expand additively, with rollback in mind. New columns nullable or defaulted; new tables and indexes freely (indexes built concurrently/online on hot tables); constraints arrive in their weakest form first (unvalidated, validated in a later step, where the engine supports it). The gate to proceed: old and new code both run clean against the expanded schema — which is exactly what a canary proves.

• Migrate with dual-writes, a bounded backfill, and a verifier. Writers write both shapes (behind a flag — ZFN-63Field Note · currentZFN-63 — Decouple deploy from release — and give every flag a death dateA deploy puts code on servers; a release changes what users see. Coupled, a deploy is a bet you can only unwind by redeploying. Decoupled by flags, deploys become boring and releases progressive and instantly reversible. But a flag is a loan: owner, death date, or Knight Capital.Why it's cited here: Deploy/release decoupling is the code-side twin: the flag that flips reads to the new column is a release decision, reversible in seconds, independent of the deploy that shipped it.Open ZFN-63 →); a backfill walks history in small, throttled, resumable batches (idempotent per batch — ZFN-19Field Note · currentZFN-19 — Annotate read-only and idempotent endpoints; make every mutation idempotentAnnotate every endpoint as read-only (safe) or idempotent, in the schema, so infrastructure can retry, route to replicas, and cache safely. Make every state-changing endpoint idempotent (idempotency keys for create/charge/send); a non-idempotent retry double-applies.Open ZFN-19 → applies to migrations too, and the primary’s health outranks the backfill’s ETA); and a comparison job proves the two shapes agree before any reader moves (Stripe’s accountOnline migrations at scale (Stripe Engineering)Stripe's worked account of migrating hundreds of millions of live objects with a four-phase dual-writing pattern — dual write, change reads, backfill and verify, remove old — while the API stayed up throughout. The migrate phase of this note, at production scale and with the verification story spelled out.stripe.com ↗ is a worked example of how much of the total effort this verification deserves: most of it). Then move readers — progressively, watching, reversibly, because reads-on-new is a release decision, not a deploy.

• Contract on evidence, on a calendar. Instrumentation says the old path is cold (query stats, column-level access logging, the dual-write flag reading 100%-new for a full business cycle — including the month-end job that grep will never find). Then: stop the dual-write, drop the old — knowing that this step is the one that genuinely breaks N−1, which is why it ships alone, after the code that needed the old shape is multiple versions gone.

• Hold the whole sequence to the compatibility law in CI. Each migration step, diffed against the previous schema, checked mechanically: no drop or rename of anything the previous release touches, no lock-taking DDL without an explicit override annotation. The review question for every migration PR is the same one: “deploy N−1 against this — what breaks?” — and the pipeline should be able to answer it by actually doing it in a staging pass.

• Keep migration execution out of application boot — a distinct pipeline stage, serialised by an advisory lock (a lease, with a named owner and a TTLField Note · currentZFN-37 — Every lock is a leaseA lock that can outlive its holder is a deadlock scheduled for later. Give every lock — including informal ones like claimed_by columns — a TTL, a named owner, and a heartbeat; make expiry automatic and server-side; fence the side effects so a stale holder can't corrupt anything.Open ZFN-37 →), forward-only (“down” migrations rehearse nicely and lie in production: the honest rollback path is the previous code, which the compatibility law guarantees still runs).

Consequences

Easier:

• Deploys and rollbacks stay boring — the schema is never the reason you can’t ship or can’t retreat, which preserves rollback as the incident remedy it must be.
• Migrations become reviewable — three small diffs with evidence gates, instead of one omnibus change whose blast radius nobody can hold in their head.
• Big-table changes stop being maintenance-window events: dual-write plus backfill plus cutover works at any size, because no step ever holds a lock the application feels.

Harder:

• Everything takes three deploys and days-to-weeks of calendar time where the atomic version took one afternoon and one outage. The afternoon was never real; the discipline makes the true cost visible and schedulable.
• The intermediate states are live machinery — dual-writes that can half-fail (ZFN-24Field Note · currentZFN-24 — One transactional store per write; propagate changes asynchronouslyCommit each logical write to exactly one transactional store; update other systems via reliable ordered async events — never a synchronous write across two stores, and never 2PC. With a relational primary the WAL is your replayable journal; write events into the same transaction.Why it's cited here: One transactional store per write is what makes this discipline tractable: one schema owns each fact, so each migration has one owner and one journal to verify against.Open ZFN-24 →: the journal-backed patterns exist precisely because two synchronous writes can disagree), flags that must be flippable, a verifier that must be believed. Sloppy middles are how migrations corrupt data slowly.
• It demands schema-change literacy in review — which DDL locks what, in your engine, at your scale, is now required knowledge for anyone approving a migration, not trivia for the DBA you don’t have.

References

• ZFN-24Field Note · currentZFN-24 — One transactional store per write; propagate changes asynchronouslyCommit each logical write to exactly one transactional store; update other systems via reliable ordered async events — never a synchronous write across two stores, and never 2PC. With a relational primary the WAL is your replayable journal; write events into the same transaction.Why it's cited here: One transactional store per write is what makes this discipline tractable: one schema owns each fact, so each migration has one owner and one journal to verify against.Open ZFN-24 → — one owner per fact; the dual-write phase is a temporary, deliberate exception that the verifier keeps honest.
• ZFN-14Field Note · currentZFN-14 — Define every API with a schema, and generate the clientsDefine every API with a machine-readable schema (OpenAPI, Protobuf, GraphQL) as the source of truth, and generate clients and server stubs from it — never hand-roll request-building and JSON parsing. Hand-written clients drift and break silently; check schema compatibility in CI.Why it's cited here: The same compatibility law one layer up: schema-first APIs get compatibility checked in CI; the database deserves the identical gate, because rollback is a consumer too.Open ZFN-14 → — the same compatibility law, same CI enforcement, one layer up.
• ZFN-63Field Note · currentZFN-63 — Decouple deploy from release — and give every flag a death dateA deploy puts code on servers; a release changes what users see. Coupled, a deploy is a bet you can only unwind by redeploying. Decoupled by flags, deploys become boring and releases progressive and instantly reversible. But a flag is a loan: owner, death date, or Knight Capital.Why it's cited here: Deploy/release decoupling is the code-side twin: the flag that flips reads to the new column is a release decision, reversible in seconds, independent of the deploy that shipped it.Open ZFN-63 → — the flag that moves readers is a release control; migrations are the first customer of deploy/release decoupling.
• ZFN-19Field Note · currentZFN-19 — Annotate read-only and idempotent endpoints; make every mutation idempotentAnnotate every endpoint as read-only (safe) or idempotent, in the schema, so infrastructure can retry, route to replicas, and cache safely. Make every state-changing endpoint idempotent (idempotency keys for create/charge/send); a non-idempotent retry double-applies.Open ZFN-19 → — backfill batches are mutations: idempotent, resumable, safe to retry.
• ParallelChangeParallelChange (Martin Fowler's bliki, by Danilo Sato)The general pattern under this note: implement a breaking interface change as expand (introduce the new alongside the old), migrate (move all consumers), then contract (remove the old) — so no single step ever breaks a consumer.martinfowler.com ↗ — the pattern; Stripe’s online migrationsOnline migrations at scale (Stripe Engineering)Stripe's worked account of migrating hundreds of millions of live objects with a four-phase dual-writing pattern — dual write, change reads, backfill and verify, remove old — while the API stayed up throughout. The migrate phase of this note, at production scale and with the verification story spelled out.stripe.com ↗ — the migrate phase at scale; gh-ostgh-ost — GitHub's online schema migration tool for MySQLA triggerless online schema-change tool: builds a shadow table, replays changes from the binlog, and cuts over when caught up. Exists because in-place DDL on large hot tables is operationally unacceptable — the tooling embodiment of 'never block writes for a schema change'.github.com ↗ — what “no locks on hot tables” looks like as tooling.

Changelog

• 2026-08-12: First published as a Field Note.