Zero-Downtime Data Migration: Backfill Race Diagram

Legacy — single store Old store only. Click Start Migration to begin the expand phase (dual-writes turn on).

Migration Control

Backfill Strategy

Live Traffic

Write rate 4.0/s

Backfill rate 10/s

Race window 0.45s

Live Readouts

Migrated

0 / 160

Consistency lag

160

Retries (self-healed)

Conflicts (data loss)

How it works

This diagrams the expand–contract pattern real teams use to migrate a live database with zero downtime.

1. Expand   — add the new store; every write
   is dual-written to old AND new (synchronous).
2. Backfill — a background job copies each
   pre-existing record from old → new.
3. Verify   — check old and new agree.
4. Contract — cut reads/writes to new only;
   retire the old store.

The bug lives in step 2. A backfill worker reads a record from the old store, then writes it into the new store after the race window below elapses. If a live dual-write updates that same record in between, which value should win?

Optimistic CAS (safe) — before writing, the backfill job checks the old value is unchanged since it was read. If a write raced it, the copy is skipped and the record is requeued for another pass (a “retry”). No data is ever lost, only delayed.
Blind overwrite (unsafe) — the backfill job writes whatever it read, no questions asked. If a write raced it, the newer value gets silently clobbered by the stale one — a real conflict / data-loss event, and that record is never revisited.

Backfill duration ≈ N / backfillRate. The race window only matters while writes and backfill overlap, so raising the write rate, lowering the backfill rate, or widening the race window slider all raise the conflict count in blind mode — try it and watch the red flashes and the conflict line climb on the chart below the grid.

Real-world relevance: this is the exact class of bug behind documented outages at companies that migrated schemas with a naive “copy everything” script instead of a version-checked backfill (GitHub, Stripe and Shopify have all written up variants of this pattern).

OLD STORENEW STORE

Old record Consistent in new Stale in new Conflict just occurred

drag to pan · scroll to zoom