Zero-downtime data migrations — the operational discipline of changing data under a running system, the practice layer above Database Internals. Why migrations break things (the five-shape taxonomy: lock queueing, resource starvation, long-transaction side effects, code-schema mismatch across rolling deploys, semantic drift — and the three vows: no data-proportional locks, reversibility until a deliberate point of no return, proof over belief), expand/contract (the grammar of all zero-downtime schema change: add alongside, migrate in compatible single-step deploys, contract late — with the per-engine fine print: concurrent index builds, declare-then-validate constraints, lock timeouts as seatbelt), backfills at scale (keyset-walked time-sized batches, throttling as a control loop yielding to replication lag, transactional checkpoints, idempotent absolute writes, the transform as reviewed code), dual-writes and shadow reads (the measurement layer: named source of truth, outbox/CDC upgrades, async sampled comparison with noise filtering, diff sweeps for cold data, idempotent repair, and the agreement rate that gates cutover), cutover and rollback (flag-controlled percentage ramps through real peaks, cohort-sticky routing, reverse dual-writes keeping rollback a flag-flip, declared windows and dated demolition), cross-store migrations (CDC as backbone — snapshot-then-stream convergence, the transform as risk concentrate, intended-equivalence verification, lag-gated cutover, pre-built reverse pipelines, and stalling as the dominant failure), and migrations as a capability (linters mechanizing the taxonomy, declarative harnesses, a control plane with ladder-climbing automation). Grounded in Wikipedia (schema migration, downtime, database schema, backward compatibility, batch processing, data migration, replication, data consistency, feature toggles, rollback, CDC, ETL, FSMs, lint).
"Run the migration at 2 a.m." was a plan when users slept somewhere. For a system that's always on, every schema change, backfill, and datastore move must happen underneath live traffic — and the ways that goes wrong are surprisingly few and surprisingly repeatable: locks that queue the world, long transactions that starve it, mismatched assumptions between old code and new schema. This series is the discipline of changing data out from under a running system; this post is the map of how it breaks.
Migration incidents come in five repeatable shapes: lock queueing, resource starvation, long-transaction side effects, code-schema mismatch across rolling deploys, and silent semantic drift. Against them, three vows: no data-proportional locks, reversibility until a deliberate late point of no return, and proof over belief. Opens the series.
Every zero-downtime schema change — rename a column, change a type, split a table, add a constraint — is the same move performed carefully: add the new shape alongside the old, migrate readers and writers in compatible steps, then remove the old shape long after anything needs it. Expand/contract (also traveling as "parallel change") is less a technique than the grammar all the techniques are written in. Learn it once and every migration becomes a sentence in it.
Expand/contract is the grammar of zero-downtime schema change: add alongside, migrate in compatible single-step deploys (write-both, read-new, write-new-only), contract late and deliberately. Plus the per-engine fine print: concurrent index builds, declare-then-validate constraints, and lock timeouts as the universal seatbelt.
The expand phase created an empty column; something has to fill it for a billion existing rows while production traffic continues oblivious. Backfills are the manual labor of migrations — conceptually trivial, operationally where migrations actually spend their calendar time and take their production risks. The craft is entirely in the adverbs: incrementally, throttled, resumably, idempotently.
Backfills are made safe by four adverbs: incrementally (keyset-walked, time-sized, per-batch-committed), throttled (a control loop yielding to replication lag, content to take days), resumably (transactional checkpoints), idempotently (absolute writes, safe re-runs). The transform is reviewed code with a verify-only mode.
Every serious migration passes through a phase where the same truth lives in two places — old column and new, old table and new, old store and new — and the entire migration's integrity rests on those copies agreeing. Hope is not a strategy and spot-checking is barely one; the disciplines that actually prove agreement are dual-writes done with eyes open, shadow reads that compare on every request, and diff sweeps that check everything else. This is the migration's measurement layer — post 1's third vow made operational.
The measurement layer: dual-writes with the race named (source of truth declared, outbox/CDC when stakes outgrow tolerate-and-repair), shadow reads comparing on live traffic with noise filtering, diff sweeps for the cold data — all feeding idempotent repair and producing the agreement rate that gates cutover.
Everything before this moment was preparation with an exit. Cutover is the moment traffic actually moves — reads, then writes, then trust — and its engineering question is blunt: how do you cross so that the first sign of trouble costs a flag-flip instead of an incident? The answer is incremental exposure, pre-staged rollback, and a disciplined patience about demolishing the bridge you just crossed.
Reads cut first (flags, percentage ramps through real peaks, cohort-sticky, canaried by the live mismatch rate); writes carry the asymmetry — reverse dual-writes keep rollback a flag-flip. Rollback is built and tested, the window declared — then the bridge is demolished on schedule, neither kept forever nor dropped early.
Same-store migrations reshape a schema; cross-store migrations change the ground itself — relational to document, single-node to distributed, self-hosted to managed, one vendor to another. Every pattern from this series still applies, but the safety nets of a single database — transactions spanning old and new, one replication stream, one consistency model — are gone, and their replacements have to be built. The backbone of those replacements is change data capture: stop writing to two places, start following one.
CDC replaces dual-writes: tail the source log, snapshot-then-stream to convergence while writes never stop. The transform is the risk concentrate; verification compares intended equivalence; cutover gates on lag with a rehearsed seconds-long pause; rollback is a pre-built reverse pipeline. The killer is stalling, not breaking.
One team running one careful migration is craft; an organization running hundreds a month across a fleet of databases is either a capability or a casino. The closing move of this series is the same one the enterprise-AI series made about adoption: stop treating each migration as a bespoke heroic project, and build the machinery that makes the safe way the easy way — linters that catch the outage before review, harnesses that embody posts 2–5 so nobody re-implements them, and automation that watches the fleet so humans watch the exceptions.
Scale migration safety by productizing it: linters that mechanize the failure taxonomy (teaching errors, signed escapes, incident-grown rules), harnesses packaging backfill/verification/cutover declaratively, and a control plane with ladder-climbing automation keeping humans on the novel. Closes the series.
This series is part of a larger body of work by Pratik Dhanave, an Agentic AI Architect writing about production AI systems, distributed systems, and cloud-native engineering. Explore all course series, browse every post, or find topics via the tag index.