Cross-Store Migrations — Changing Databases Entirely
Same-store migrations reshape a schema; cross-store migrations change the ground itself — relational to document, single-node to distributed, self-hosted to managed, one vendor to another. Every pattern from this series still applies, but the safety nets of a single database — transactions spanning old and new, one replication stream, one consistency model — are gone, and their replacements have to be built. The backbone of those replacements is change data capture: stop writing to two places, start following one.
Posts 2–5 built the method inside one store; this post rebuilds it across two — the territory of replatforming projects, with the CDC machinery that Stream Processing and the Spanner migration tooling I’ve contributed to both live in.
CDC replaces dual-writes as the backbone
Post 4 flagged the cross-store dual-write problem: two writes, no shared transaction, divergence by design. The structural fix is to stop having two writers. Change data capture tails the source database’s own write-ahead/replication log and emits every committed change as an ordered stream; a pipeline applies that stream to the target. One writer (production, unchanged), one follower (the pipeline) — the two-generals gap replaced by a consumer offset. The properties that make CDC the backbone: it captures what actually committed (no app-level race can diverge it), it carries ordering per key (the log’s order), it’s resumable (offsets checkpoint like post 3’s cursors), and it keeps capturing during the long initial copy, which solves the hardest sequencing problem — snapshot-then-stream: bulk-copy a consistent snapshot (post 3’s batched machinery, pointed across stores), note the log position the snapshot corresponds to, then apply the stream from that position; the target converges to live without a write ever stopping.
The pipeline inherits every verb this series taught: applies are idempotent (replays happen), throttled (the target has its own production neighbors), checkpointed, observed — with one new verb: transform. Cross-store means the shapes differ — rows to documents, foreign keys to embedded structures, types to their nearest cousins — and the transform carries all of post 1’s semantic-drift risk concentrated in one component. It gets the full treatment: reviewed as production code, property-tested on the nasty values (encodings, timezones, nulls, numeric edges), and judged by verification rather than by passing its own tests.
Verification and cutover, harder edition
Post 4’s instruments return with cross-store complications. Comparison needs a common language — you can’t checksum a Postgres range against a DynamoDB partition directly; the diff sweep compares canonicalized representations (apply the transform to the source, compare transformed-source to target), which conveniently also tests the transform continuously. Shadow reads cross a semantic gap — the new store may return differently-ordered sets, different precision, different null semantics; the comparator must encode intended equivalence, and every “that difference is fine” decision gets written down, because those decisions are the migration’s real spec. Lag is now first-class — CDC lag is the staleness of the new store; read-cutover ramps (post 5) gate on lag bounds, and write cutover requires lag ≈ 0 at the moment of flip (a brief write pause — seconds of queued writes, not downtime — is the honest implementation of the final switch in most real systems; design it, rehearse it, measure it). Rollback means reverse CDC — keeping the bridge (post 5) now means a return pipeline from new store back to old, built and tested before write cutover, because improvising a reverse replication path during an incident is not a plan. And the dual-stack period costs real money and attention — two databases, two on-call surfaces, a pipeline between — which is why cross-store migrations fail more often by stalling (half-migrated for a year) than by breaking; the declared rollback window and dated demolition from post 5 matter double here.
The takeaway: cross-store migration is the series’ method with the shared-database safety nets rebuilt — CDC as the backbone (one writer, log-ordered follower, snapshot-then-stream convergence), the transform as the concentrated-risk component (reviewed, property-tested, continuously judged by canonicalized diffing), verification comparing intended equivalence with the fine-print decisions written down, cutover gated on CDC lag with a rehearsed seconds-long write pause at the flip, and rollback as a pre-built reverse pipeline. The dominant failure mode isn’t the outage — it’s the stall; the cure is the same declared windows and dated exits as ever, enforced against a dual-stack that costs money every week it lives.
Key takeaways
- Replace dual-writes with CDC: tail the source’s log — committed truth, per-key ordering, resumable offsets — and snapshot-then-stream from a noted log position so the target converges while writes never stop.
- The transform is the migration’s risk concentrate: production-code review, property tests on encodings/timezones/nulls/numeric edges, and continuous judgment via canonicalized source-vs-target diffing.
- Verification crosses a semantic gap: comparators encode intended equivalence, and every tolerated difference is documented — that document is the real spec.
- Cutover gates on CDC lag; the final write flip is honestly implemented as a rehearsed, seconds-long pause; rollback = a reverse CDC pipeline built before it’s needed.
- Cross-store migrations die by stalling more than breaking — dual-stack costs weekly; declared windows and dated demolition are the cure, doubly.
Further reading
- Change data capture — following a database’s own log of truth
- Extract, transform, load — the bulk-movement discipline the snapshot phase borrows