Minimal-Downtime Spanner Migration

A long bulk load runs alongside change data capture streamed through Datastream, Pub/Sub, and Dataflow, then a quick traffic cutover

Minimal-Downtime Spanner Migration A long bulk load runs alongside change data capture streamed through Datastream, Pub/Sub, and Dataflow, then a quick traffic cutover 01 / Source 02 / Capture 03 / Transport 04 / Apply 05 / Cutover Source DB · live production · 01 / Source · read + write Source DB live production read + write Bulk Load · hours-long snapshot · 02 / Capture · one-time Bulk Load hours-long snapshot one-time Datastream · change capture · 02 / Capture · CDC Datastream change capture CDC Pub/Sub · change events · 03 / Transport · buffered Pub/Sub change events buffered Dataflow · apply deltas · 04 / Apply · streaming Dataflow apply deltas streaming Cloud Spanner · destination · 05 / Cutover · target Cloud Spanner destination target Traffic Cutover · quick switch · 05 / Cutover · seconds Traffic Cutover quick switch seconds snapshot read change log initial load CDC events stream upserts caught up Legend primary data policy / PII async batch data store

Two Parallel Paths

  • • A one-time bulk load seeds Spanner over hours
  • • CDC streams every live change at the same time
  • • The destination converges on the source with no freeze

Streaming Pipeline

  • • Datastream captures the source change log
  • • Pub/Sub buffers change events for replay
  • • Dataflow applies deltas as idempotent upserts

Minimal Downtime

  • • Cutover waits until replication lag nears zero
  • • Traffic switches in seconds, not hours
  • • Rollback stays possible until the switch is final

Data-flow diagram • Built with Archify • Create yours ↗ • Hover to trace • R route • Click to focus • +/− zoom • M radar • [/] views • P play story • T theme • E export