#Migrations

Articles about Migrations — exploring patterns, best practices, and real-world implementations in production systems.

8 posts tagged with migrations. ← Blog home

#A/B Testing (3)#A2A (14)#ADK (8)#AG-UI (6)#AI (9)#AI Adoption (9)#AI Agents (355)#AI Architecture (22)#AI Cost (10)#AI Cost Optimization (8)#AI Engineering (316)#AI Evaluation (9)#AI Financial Engineering (14)#AI Gateway (8)#AI Governance (43)#AI Red Teaming (9)#AI Research (17)#AI Safety (12)#AI Security (35)#AI Strategy (8)#AI in Production (12)#AML (3)#API Design (17)#API Security (9)#APIs (69)#AWS (17)#Accountability (8)#Accounting (9)#Agent Memory (8)#Agent Skills (11)#Agentic AI (25)#Agentic Commerce (12)#Agentic Engineering (12)#Agentic RAG (8)#Agents (4)#Amazon Bedrock (8)#Analytics (3)#Architecture (45)#Attention (3)#Audit (4)#Authentication (11)#Authorization (3)#Automation (13)#Azure (11)#Azure AI Foundry (9)#Backend Engineering (322)#Benchmarks (3)#Best Practices (5)#BigQuery (6)#Billing (8)#Build Systems (4)#Business Finance (8)#Business Strategy (55)#C (8)#CI/CD (23)#Caching (11)#Capital Markets (14)#Card Payments (12)#Cards (13)#Career (30)#Causal Inference (8)#Cellular Automata (8)#Change Management (3)#Checkpointing (4)#Claude Code (8)#Cloud (5)#Cloud Architecture (3)#Cloud Native (10)#Code Review (9)#Coding Agents (20)#Collaboration (6)#Communication (9)#Complexity (8)#Compliance (54)#Computer Networking (9)#Computer Science (40)#Computer Vision (15)#Computer-Use Agents (8)#Concurrency (42)#Consulting (26)#Containers (10)#Context Engineering (19)#Conversational AI (8)#Cost Optimisation (5)#Cost Optimization (3)#Credit (16)#Credit Risk (16)#CrewAI (8)#Crypto (12)#Cryptocurrency (12)#Cryptography (9)#Custody (9)#DSPy (8)#Data (14)#Data Engineering (25)#Data Governance (3)#Data Science (8)#Data Structures (10)#Databases (48)#Deep Learning (20)#Deployment (5)#Design (3)#Design Patterns (10)#DevOps (24)#DevSecOps (22)#Developer Experience (7)#Developer Productivity (15)#Developer Tools (5)#Differential Privacy (3)#Diffusion Models (10)#Discovery (5)#Distributed Systems (104)#Documentation (3)#Edge AI (8)#Embeddings (21)#Emotional Intelligence (8)#Energy (8)#Engineering (14)#Engineering Culture (3)#Engineering Judgment (7)#Engineering Practices (21)#Enterprise AI (8)#Error Handling (4)#Evaluation (79)#Event-Driven (9)#Event-Driven Architecture (8)#Explainability (3)#Explainable AI (8)#FREE-AI (8)#FX (5)#Feature Engineering (3)#Feature Flags (3)#Feedback (4)#FinOps (23)#FinTech (6)#Financial AI (14)#Financial Engineering (8)#Financial Systems (133)#Fine-Tuning (12)#Fintech (134)#Flutter (8)#Forecasting (10)#Foreign Exchange (5)#Forward Deployed Engineer (43)#Forward Deployed Engineering (8)#Fraud (10)#Fraud Detection (8)#Function Tools (5)#Functional Programming (3)#Fundamentals (23)#Fundraising (8)#GCP (5)#GPU (8)#GUI Automation (8)#Gemma (4)#Generative AI (13)#Git (8)#Go (232)#Go-to-Market (31)#Google ADK (36)#Governance (61)#Granite (6)#Graph Neural Networks (8)#GraphQL (3)#Growth (3)#Guardrails (35)#HIPAA (3)#HTTP (3)#Harness Engineering (8)#Hiring (9)#Hugging Face (8)#Human-in-the-Loop (10)#IBM watsonx (8)#Idempotency (3)#Identity (11)#Image Generation (10)#Inference (8)#Information Retrieval (8)#Integration (3)#Integrations (8)#Intellectual Property (8)#Interfaces (3)#Interpretability (8)#JavaScript (8)#KYC (11)#KYC and AML (12)#Kafka (11)#Knowledge Graphs (11)#Kubernetes (17)#LLM (5)#LLM Inference (8)#LLM Infrastructure (8)#LLM-as-Judge (3)#LLMOps (9)#LLMs (197)#LangChain (8)#LangGraph (11)#Leadership (33)#Ledger (12)#Legal (8)#Lending (14)#Linux (9)#LlamaIndex (8)#Load Balancing (3)#MCP (25)#MLOps (39)#Machine Learning (119)#Marketing (16)#Markets (4)#Mathematical Finance (8)#Memory (16)#Memory Management (5)#Metrics (7)#Microservices (3)#Microsoft Agent Framework (150)#Middleware (6)#Migration (9)#Migrations (8)#Mixture of Experts (5)#Model Risk (3)#Monetization (9)#Monitoring (5)#Monorepo (3)#Multi-Agent (10)#Multi-Agent AI (14)#Multi-Agent Systems (76)#Multimodal (3)#Multimodal AI (11)#NIM (5)#NVIDIA (8)#Networking (3)#OAuth (3)#OWASP (7)#Observability (53)#On-Device AI (8)#Ontology (8)#Open Source (7)#OpenAPI (3)#OpenTelemetry (5)#Operating Systems (9)#Operations (14)#Opinion (6)#Orchestration (10)#Organizational Design (8)#Payment Rails (16)#Payments (55)#People (8)#Performance (56)#Personalization (9)#Platform Engineering (17)#Policy as Code (3)#PreSales (8)#Pricing (4)#Privacy (14)#Privacy Engineering (3)#Process (4)#Product (30)#Product Management (9)#Production (22)#Programming (10)#Programming Languages (48)#Prompt Engineering (86)#Prompt Injection (16)#Proof of Concept (3)#Protocol Buffers (3)#Protocols (9)#Providers (4)#Pydantic AI (8)#Python (142)#Quality (3)#Quantitative Finance (22)#RAG (72)#RBI (3)#REST (5)#Rails (16)#Real-Time (8)#Reasoning Models (8)#Recommender Systems (8)#Reconciliation (5)#RegTech (8)#Regulation (10)#Reinforcement Learning (9)#Reliability (68)#Resilience (4)#Responsible AI (5)#Retrieval (3)#Risk (22)#Risk Management (11)#Rust (32)#SLSA (3)#SRE (23)#SaaS (9)#Sales (9)#Sales Engineering (8)#Scalability (6)#Search (10)#Security (103)#Security Engineering (8)#Self-Evolving Agents (24)#Semantic Web (8)#Sessions (3)#Settlement (9)#Soft Skills (8)#Software (3)#Software Architecture (36)#Software Delivery (9)#Software Engineering (152)#Spanner (4)#Specification Engineering (3)#Speech (8)#Startups (30)#Statistics (8)#Strands (8)#Strategy (6)#Stream Processing (9)#Streaming (38)#Structured Output (4)#Supply Chain Security (10)#Sustainability (9)#System Design (32)#Systems Programming (56)#Technical Sales (6)#Testing (79)#Threat Modeling (3)#Time Series (9)#Tool Use (23)#Tooling (11)#Tools (4)#Trading (8)#Transformers (3)#Treasury (6)#Trust (4)#Type System (3)#Type Systems (12)#TypeScript (8)#Validation (3)#Valuation (3)#Vector Databases (23)#Vector Search (12)#Venture Capital (8)#Version Control (9)#Versioning (7)#Voice AI (9)#Web Development (6)#Webhooks (9)#Workflow (4)#Workflows (14)#eBPF (8)#gRPC (13)#smolagents (8)
Pratik Dhanave · ·4 min read

Migrations as a Capability — Linters, Harnesses, and the Fleet View

One team running one careful migration is craft; an organization running hundreds a month across a fleet of databases is either a capability or a casino. The closing move of this series is the same one the enterprise-AI series made about adoption: stop treating each migration as a bespoke heroic project, and build the machinery that makes the safe way the easy way — linters that catch the outage before review, harnesses that embody posts 2–5 so nobody re-implements them, and automation that watches the fleet so humans watch the exceptions.

Scale migration safety by productizing it: linters that mechanize the failure taxonomy (teaching errors, signed escapes, incident-grown rules), harnesses packaging backfill/verification/cutover declaratively, and a control plane with ladder-climbing automation keeping humans on the novel. Closes the series.

#Migrations#Platform Engineering#Automation#Databases#Reliability
Pratik Dhanave · ·4 min read

Cross-Store Migrations — Changing Databases Entirely

Same-store migrations reshape a schema; cross-store migrations change the ground itself — relational to document, single-node to distributed, self-hosted to managed, one vendor to another. Every pattern from this series still applies, but the safety nets of a single database — transactions spanning old and new, one replication stream, one consistency model — are gone, and their replacements have to be built. The backbone of those replacements is change data capture: stop writing to two places, start following one.

CDC replaces dual-writes: tail the source log, snapshot-then-stream to convergence while writes never stop. The transform is the risk concentrate; verification compares intended equivalence; cutover gates on lag with a rehearsed seconds-long pause; rollback is a pre-built reverse pipeline. The killer is stalling, not breaking.

#Migrations#CDC#Databases#Replatforming#Reliability
Pratik Dhanave · ·4 min read

Cutover and Rollback — Crossing Carefully, and Keeping the Bridge

Everything before this moment was preparation with an exit. Cutover is the moment traffic actually moves — reads, then writes, then trust — and its engineering question is blunt: how do you cross so that the first sign of trouble costs a flag-flip instead of an incident? The answer is incremental exposure, pre-staged rollback, and a disciplined patience about demolishing the bridge you just crossed.

Reads cut first (flags, percentage ramps through real peaks, cohort-sticky, canaried by the live mismatch rate); writes carry the asymmetry — reverse dual-writes keep rollback a flag-flip. Rollback is built and tested, the window declared — then the bridge is demolished on schedule, neither kept forever nor dropped early.

#Migrations#Feature Flags#Reliability#Databases
Pratik Dhanave · ·4 min read

Dual-Writes and Shadow Reads — Proving Two Copies Agree

Every serious migration passes through a phase where the same truth lives in two places — old column and new, old table and new, old store and new — and the entire migration's integrity rests on those copies agreeing. Hope is not a strategy and spot-checking is barely one; the disciplines that actually prove agreement are dual-writes done with eyes open, shadow reads that compare on every request, and diff sweeps that check everything else. This is the migration's measurement layer — post 1's third vow made operational.

The measurement layer: dual-writes with the race named (source of truth declared, outbox/CDC when stakes outgrow tolerate-and-repair), shadow reads comparing on live traffic with noise filtering, diff sweeps for the cold data — all feeding idempotent repair and producing the agreement rate that gates cutover.

#Migrations#Verification#Databases#Reliability
Pratik Dhanave · ·4 min read

Gradual Typing Migrations — Retrofitting Types onto a Living Codebase

The great typing migrations — millions of dynamic-language lines brought under TypeScript, or type checkers built for Ruby and Python — are the canonical org-loop projects: multi-year, codebase-wide, impossible to pause the business for, and transformative when they land. They're worth studying even if you never run one, because they exercise every DevProd muscle at once: incremental adoption machinery, codemod fleets, ratchets that prevent backsliding, and the political craft of keeping a thousand engineers on board for years.

The worked example of org-loop transformation: a strictness map turned module by module, explicit boundaries with counted escape hatches, trunk-first sequencing — run on inference-powered codemod fleets, a CI ratchet making typedness monotonic, and the political craft that keeps a thousand engineers on board for years.

#Developer Productivity#Type Systems#Migrations#Tooling#Platform Engineering
Pratik Dhanave · ·4 min read

Backfills at Scale — Rewriting a Billion Rows Under Live Traffic

The expand phase created an empty column; something has to fill it for a billion existing rows while production traffic continues oblivious. Backfills are the manual labor of migrations — conceptually trivial, operationally where migrations actually spend their calendar time and take their production risks. The craft is entirely in the adverbs: incrementally, throttled, resumably, idempotently.

Backfills are made safe by four adverbs: incrementally (keyset-walked, time-sized, per-batch-committed), throttled (a control loop yielding to replication lag, content to take days), resumably (transactional checkpoints), idempotently (absolute writes, safe re-runs). The transform is reviewed code with a verify-only mode.

#Migrations#Databases#Backfills#Reliability
Pratik Dhanave · ·4 min read

Expand/Contract — The Only Schema Change Pattern You Need

Every zero-downtime schema change — rename a column, change a type, split a table, add a constraint — is the same move performed carefully: add the new shape alongside the old, migrate readers and writers in compatible steps, then remove the old shape long after anything needs it. Expand/contract (also traveling as "parallel change") is less a technique than the grammar all the techniques are written in. Learn it once and every migration becomes a sentence in it.

Expand/contract is the grammar of zero-downtime schema change: add alongside, migrate in compatible single-step deploys (write-both, read-new, write-new-only), contract late and deliberately. Plus the per-engine fine print: concurrent index builds, declare-then-validate constraints, and lock timeouts as the universal seatbelt.

#Migrations#Databases#Schema Changes#Reliability#Concurrency
Pratik Dhanave · ·4 min read

Why Migrations Break Things — The Downtime Taxonomy

"Run the migration at 2 a.m." was a plan when users slept somewhere. For a system that's always on, every schema change, backfill, and datastore move must happen underneath live traffic — and the ways that goes wrong are surprisingly few and surprisingly repeatable: locks that queue the world, long transactions that starve it, mismatched assumptions between old code and new schema. This series is the discipline of changing data out from under a running system; this post is the map of how it breaks.

Migration incidents come in five repeatable shapes: lock queueing, resource starvation, long-transaction side effects, code-schema mismatch across rolling deploys, and silent semantic drift. Against them, three vows: no data-proportional locks, reversibility until a deliberate late point of no return, and proof over belief. Opens the series.

#Migrations#Databases#Reliability#Schema Changes

All posts on this site are written by Pratik Dhanave, an Agentic AI Architect with 7+ years building production distributed systems, multi-agent AI platforms, and cloud-native infrastructure. About the author → Each article includes working code, architecture diagrams, and references to the specific frameworks and standards discussed. Browse the blog or explore related topics using the tag cloud above.