---
id: 62
title: "Expand, migrate, contract: schema changes in three moves"
kind: note
status: current
date: 2026-08-12
authors:
  - "Theo Zourzouvillys"
tags: [data, architecture, reliability, operations, deploy]
references:
  - id: parallelchange
    title: "ParallelChange (Martin Fowler's bliki, by Danilo Sato)"
    url: https://martinfowler.com/bliki/ParallelChange.html
    abstract: "The general pattern under this note: implement a breaking interface change as expand (introduce the new alongside the old), migrate (move all consumers), then contract (remove the old) — so no single step ever breaks a consumer."
  - id: stripemigrations
    title: "Online migrations at scale (Stripe Engineering)"
    url: https://stripe.com/blog/online-migrations
    abstract: "Stripe's worked account of migrating hundreds of millions of live objects with a four-phase dual-writing pattern — dual write, change reads, backfill and verify, remove old — while the API stayed up throughout. The migrate phase of this note, at production scale and with the verification story spelled out."
  - id: ghost
    title: "gh-ost — GitHub's online schema migration tool for MySQL"
    url: https://github.com/github/gh-ost
    abstract: "A triggerless online schema-change tool: builds a shadow table, replays changes from the binlog, and cuts over when caught up. Exists because in-place DDL on large hot tables is operationally unacceptable — the tooling embodiment of 'never block writes for a schema change'."
summary: "Every deploy runs two code versions against one database — and rollback runs yesterday's code on today's schema. No schema change may break either. So every migration is three shippable moves: expand (additive), migrate (backfill, verify), contract (remove, later, deliberately)."
supersedes: null
superseded_by: null
aliases: []
crossrefs:
  ZFN-24: "One transactional store per write is what makes this discipline tractable: one schema owns each fact, so each migration has one owner and one journal to verify against."
  ZFN-14: "The same compatibility law one layer up: schema-first APIs get compatibility checked in CI; the database deserves the identical gate, because rollback is a consumer too."
  ZFN-63: "Deploy/release decoupling is the code-side twin: the flag that flips reads to the new column is a release decision, reversible in seconds, independent of the deploy that shipped it."
---

## TL;DR

**During every deploy, two versions of your code run against one database — and if you ever
roll back, yesterday's code runs against today's schema. Both must work. That single fact
generates the entire discipline:**

- **Every schema change decomposes into three independently-shippable moves**
  ([ParallelChange](ref:parallelchange)):
  1. **Expand** — purely additive: new column (nullable or defaulted), new table, new index.
     Old code doesn't notice.
  2. **Migrate** — dual-write, backfill in bounded batches, **verify by comparison**, move
     readers over ([Stripe's online-migration pattern](ref:stripemigrations)).
  3. **Contract** — drop the old column/table/path, **only after evidence** (instrumentation,
     not grep) says nothing reads or writes it.
- **Never rename in place, never change a type in place, never add NOT NULL to a live column
  in one step.** Each of those is the three moves compressed into one breaking one.
- **Schema migrations deploy like code but are not coupled to code deploys** — never run from
  app boot, always forward-only, each step compatible one version in both directions.
- **Contract is a scheduled task, not an aspiration** — or your schema becomes an archaeology
  of half-finished migrations.

## Context

The deploy-time overlap isn't an edge case you can squeeze out: rolling deploys, canaries,
[draining instances still serving during shutdown](/zfn/60-graceful-shutdown-is-a-protocol/) —
heterogeneous code against one schema is the *normal* operating condition of a continuously
deployed system. And the rollback case is stricter and always forgotten: rollback is your
primary incident remedy, it must stay cheap ([ZFN-63](/zfn/63-decouple-deploy-from-release/)),
and it means *last week's* code meeting *this week's* schema. A schema change that breaks
version N−1 hasn't just constrained the deploy — it has quietly disabled the undo button.

What goes wrong without the discipline:

- **The atomic rename.** `ALTER TABLE … RENAME COLUMN` plus the code change in one release:
  every not-yet-updated instance breaks the moment the DDL commits, and rollback breaks the
  updated ones. The outage window is the deploy window, and the only exit is forward.
- **The innocent-looking lock.** DDL that looks instant can take a table lock and rewrite or
  scan the whole thing — behind it, every query queues, the connection pool fills, and the app
  is down without a single error in the migration log. Engines differ on *which* operations
  bite (and versions keep improving it); the rule that doesn't change is: **know the locking
  behaviour of every DDL statement you run against a hot table, in your engine, at your
  size** — and where in-place is unacceptable, use the shadow-table machinery
  ([gh-ost](ref:ghost) and friends) built for exactly this.
- **The app-boot migration.** Migrations run by whichever instance boots first couple schema
  change to deploy topology: N instances race, a slow migration makes deploy timeouts kill
  half-applied work, and rollback re-runs nothing. Migration execution is its own pipeline
  step with its own logs, locks, and human-visible outcome.

The through-line: **a database schema is an API with the strictest compatibility contract you
operate** — its consumers can't be simultaneously updated, ever. You already hold APIs to
one-version compatibility with CI-enforced checks
([ZFN-14](/zfn/14-schema-first-apis-generate-clients/)); the schema deserves the same law, with
rollback counted as a consumer.

> [!aside]
>
> The contract step is where migrations go to die. Expand and migrate ship because a feature
> needs them; contract ships because someone remembered. Two years later the table has both
> `email` and `email_address`, three "temporary" dual-write paths, and a NOT NULL constraint
> nobody dares add — every future migration now costs double because the last five never
> finished. Book the contract step into the tracker *when the expand step merges*, or accept
> that you're choosing the archaeology.

## Recommendation

**Run every schema change as three reviewed, independently-deployed moves — with evidence
gates between them.**

- **Expand additively, with rollback in mind.** New columns nullable or defaulted; new tables
  and indexes freely (indexes built concurrently/online on hot tables); constraints arrive in
  their weakest form first (unvalidated, validated in a later step, where the engine supports
  it). The gate to proceed: old and new code both run clean against the expanded schema — which
  is exactly what a canary proves.

- **Migrate with dual-writes, a bounded backfill, and a verifier.** Writers write both shapes
  (behind a flag — [ZFN-63](/zfn/63-decouple-deploy-from-release/)); a backfill walks history
  in small, throttled, resumable batches (idempotent per batch —
  [ZFN-19](/zfn/19-annotate-readonly-idempotent-endpoints/) applies to migrations too, and the
  primary's health outranks the backfill's ETA); and a **comparison job proves the two shapes
  agree before any reader moves** ([Stripe's account](ref:stripemigrations) is a worked
  example of how much of the total effort this verification deserves: most of it). Then move
  readers — progressively, watching, reversibly, because reads-on-new is a release decision,
  not a deploy.

- **Contract on evidence, on a calendar.** Instrumentation says the old path is cold (query
  stats, column-level access logging, the dual-write flag reading 100%-new for a full business
  cycle — including the month-end job that grep will never find). Then: stop the dual-write,
  drop the old — knowing that *this* step is the one that genuinely breaks N−1, which is why
  it ships alone, after the code that needed the old shape is multiple versions gone.

- **Hold the whole sequence to the compatibility law in CI.** Each migration step, diffed
  against the previous schema, checked mechanically: no drop or rename of anything the
  previous release touches, no lock-taking DDL without an explicit override annotation. The
  review question for every migration PR is the same one:
  *"deploy N−1 against this — what breaks?"* — and the pipeline should be able to answer it
  by actually doing it in a staging pass.

- **Keep migration execution out of application boot** — a distinct pipeline stage, serialised
  by an advisory lock ([a lease, with a named owner and a TTL](/zfn/37-every-lock-is-a-lease/)),
  forward-only ("down" migrations rehearse nicely and lie in production: the honest rollback
  path is the *previous code*, which the compatibility law guarantees still runs).

## Consequences

**Easier:**

- **Deploys and rollbacks stay boring** — the schema is never the reason you can't ship or
  can't retreat, which preserves rollback as the incident remedy it must be.
- **Migrations become reviewable** — three small diffs with evidence gates, instead of one
  omnibus change whose blast radius nobody can hold in their head.
- **Big-table changes stop being maintenance-window events**: dual-write plus backfill plus
  cutover works at any size, because no step ever holds a lock the application feels.

**Harder:**

- **Everything takes three deploys and days-to-weeks of calendar time** where the atomic
  version took one afternoon and one outage. The afternoon was never real; the discipline
  makes the true cost visible and schedulable.
- **The intermediate states are live machinery** — dual-writes that can half-fail
  ([ZFN-24](/zfn/24-one-transactional-store-per-write/): the journal-backed patterns exist
  precisely because two synchronous writes can disagree), flags that must be flippable, a
  verifier that must be believed. Sloppy middles are how migrations corrupt data *slowly*.
- **It demands schema-change literacy in review** — which DDL locks what, in your engine, at
  your scale, is now required knowledge for anyone approving a migration, not trivia for the
  DBA you don't have.

## References

- [ZFN-24](/zfn/24-one-transactional-store-per-write/) — one owner per fact; the dual-write
  phase is a temporary, deliberate exception that the verifier keeps honest.
- [ZFN-14](/zfn/14-schema-first-apis-generate-clients/) — the same compatibility law, same CI
  enforcement, one layer up.
- [ZFN-63](/zfn/63-decouple-deploy-from-release/) — the flag that moves readers is a release
  control; migrations are the first customer of deploy/release decoupling.
- [ZFN-19](/zfn/19-annotate-readonly-idempotent-endpoints/) — backfill batches are mutations:
  idempotent, resumable, safe to retry.
- [ParallelChange](ref:parallelchange) — the pattern; [Stripe's online
  migrations](ref:stripemigrations) — the migrate phase at scale;
  [gh-ost](ref:ghost) — what "no locks on hot tables" looks like as tooling.

## Changelog

- **2026-08-12**: First published as a Field Note.