Zero-Downtime Infrastructure Migration Strategies

Introduction

Zero-downtime migration means moving infrastructure, databases, or applications without users experiencing any service interruption. This is one of the most challenging problems in production engineering. A junior engineer often thinks "we'll just take a maintenance window" — but in 24/7 global services, there is no acceptable maintenance window. This guide teaches you the patterns and techniques to migrate infrastructure without downtime.

The Blue-Green Deployment Pattern

Blue-green is the simplest zero-downtime strategy: maintain two identical environments (blue = current, green = new), switch traffic when green is ready.

TEXT

Before:                    During migration:              After:

Internet → LB → Blue       Internet → LB → Blue(90%)    Internet → LB → Green
                                              Green(10%)

BASH

# Example: Blue-Green with nginx upstream
# /etc/nginx/conf.d/app.conf

upstream backend_blue {
    server 10.0.1.10:8080;
    server 10.0.1.11:8080;
}

upstream backend_green {
    server 10.0.2.10:8080;
    server 10.0.2.11:8080;
}

# Currently routing to blue
upstream backend {
    server 10.0.1.10:8080;
    server 10.0.1.11:8080;
}

server {
    listen 80;
    location / {
        proxy_pass http://backend;
    }
}

# Switch to green: update upstream and reload
# nginx -t && nginx -s reload   (zero-downtime reload)

Database Migration Without Downtime

Database schema changes are the hardest part of zero-downtime deployments. The key: expand-contract pattern.

Phase 1 - Expand (deploy while old app runs):

SQL

-- Add new column (nullable, so old app doesn't need it)
ALTER TABLE users ADD COLUMN phone_e164 VARCHAR(20) NULL;

-- Create index concurrently (doesn't lock table in PostgreSQL)
CREATE INDEX CONCURRENTLY idx_users_phone_e164 ON users(phone_e164);

Phase 2 - Migrate data (backfill without locking):

PYTHON

#!/usr/bin/env python3
# backfill_phone.py - Migrate in small batches to avoid table locks
import psycopg2
import time

conn = psycopg2.connect("dbname=prod user=app")
cur = conn.cursor()

batch_size = 1000
last_id = 0

while True:
    cur.execute('''
        UPDATE users
        SET phone_e164 = normalize_phone(phone)
        WHERE id > %s
          AND phone_e164 IS NULL
          AND phone IS NOT NULL
        ORDER BY id
        LIMIT %s
        RETURNING id
    ''', (last_id, batch_size))

    rows = cur.fetchall()
    if not rows:
        break

    last_id = max(r[0] for r in rows)
    conn.commit()
    print(f"Migrated {len(rows)} rows, last id: {last_id}")
    time.sleep(0.1)  # Throttle: give database breathing room

Phase 3 - Deploy new app (reads both columns, writes new column):

PYTHON

# Application reads phone_e164, falls back to old phone
def get_user_phone(user):
    return user.phone_e164 or normalize_phone(user.phone)

Phase 4 - Contract (after verifying migration):

SQL

-- Add NOT NULL constraint only after all rows filled
ALTER TABLE users ALTER COLUMN phone_e164 SET NOT NULL;

-- Drop old column later (separate deployment)
-- ALTER TABLE users DROP COLUMN phone;

Rolling Updates in Kubernetes

YAML

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0    # Never reduce below 10 running pods
      maxSurge: 2          # Allow up to 12 pods during update

  template:
    spec:
      containers:
      - name: web
        image: company/web:v2.0
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
          failureThreshold: 3
        # App must pass readiness before receiving traffic

BASH

# Deploy and watch the rollout
kubectl set image deployment/web-app web=company/web:v2.1
kubectl rollout status deployment/web-app --timeout=5m

# If something goes wrong, rollback instantly
kubectl rollout undo deployment/web-app

# Check rollout history
kubectl rollout history deployment/web-app

Traffic Shifting with Canary Releases

YAML

# Canary: send 5% of traffic to new version
# Using Argo Rollouts for sophisticated canary
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: web-app
spec:
  strategy:
    canary:
      steps:
      - setWeight: 5     # 5% to canary
      - pause: {duration: 10m}
      - setWeight: 20    # 20% if no errors
      - pause: {duration: 10m}
      - setWeight: 50
      - pause: {duration: 5m}
      - setWeight: 100   # Full rollout
      analysis:
        templates:
        - templateName: error-rate
        startingStep: 1
        args:
        - name: service-name
          value: web-app-canary

Live Service Migration (Stateful)

Migrating a stateful service (like a database) with replication:

BASH

# Example: Migrate MySQL to new server without downtime

# Step 1: Set up replication to new server
# On new server:
mysql -e "CHANGE MASTER TO
  MASTER_HOST='old-db.company.com',
  MASTER_USER='replication',
  MASTER_PASSWORD='replpass',
  MASTER_AUTO_POSITION=1;"
mysql -e "START SLAVE;"

# Step 2: Monitor replication lag
watch -n 5 "mysql -e 'SHOW SLAVE STATUSG' | grep Seconds_Behind_Master"

# Step 3: When lag = 0, switch application to new server
# In app config: DB_HOST=new-db.company.com

# Step 4: After switch confirmed, stop old server

Health Checks and Graceful Shutdown

Applications must support graceful shutdown to avoid in-flight request drops:

PYTHON

# app.py - Flask app with graceful shutdown
from flask import Flask
import signal
import time
import threading

app = Flask(__name__)
shutdown_requested = False

@app.route('/health')
def health():
    if shutdown_requested:
        return 'shutting down', 503
    return 'ok', 200

@app.route('/api/data')
def data():
    # Simulate request processing
    time.sleep(0.1)
    return {'data': 'result'}

def handle_sigterm(signum, frame):
    global shutdown_requested
    shutdown_requested = True
    # Give in-flight requests 30 seconds to complete
    print("SIGTERM received, waiting 30s for in-flight requests...")
    time.sleep(30)

signal.signal(signal.SIGTERM, handle_sigterm)

In Kubernetes, configure terminationGracePeriodSeconds:

YAML

spec:
  terminationGracePeriodSeconds: 60
  containers:
  - name: app
    lifecycle:
      preStop:
        exec:
          command: ["/bin/sh", "-c", "sleep 5"]  # Wait for LB to stop sending traffic

Testing Zero-Downtime Migrations

BASH

#!/bin/bash
# test_zero_downtime.sh - Run continuous requests during deployment
ERRORS=0
TOTAL=0

while true; do
    HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" https://app.company.com/api/health)
    TOTAL=$((TOTAL + 1))
    if [ "$HTTP_CODE" != "200" ]; then
        ERRORS=$((ERRORS + 1))
        echo "ERROR at $(date): HTTP $HTTP_CODE"
    fi
    sleep 0.1
done

# Run in background while you deploy:
# ./test_zero_downtime.sh &
# kubectl set image deployment/app app=company/app:v2
# # When done:
# kill %1
# echo "Total: $TOTAL, Errors: $ERRORS"

Zero-downtime migrations require careful planning, proper health checks, and incremental traffic shifting. Start with blue-green for stateless services, use the expand-contract pattern for databases, and always test your rollback procedure before executing the migration.