This models the expand–contract pattern real teams use to migrate a live database with zero downtime — the same "data migration" step the article groups with platform and architecture migration.
1. Expand — add the new store; every write
is dual-written to old AND new (synchronous).
2. Backfill — a background job copies each
pre-existing record from old → new.
3. Verify — check old and new agree.
4. Contract — cut reads/writes to new only;
retire the old store.
The bug lives in step 2. A backfill worker reads a record from the old store, then writes it into the new store a moment later. If a live dual-write updates that same record in between, which value should win?
- Optimistic CAS (safe) — before writing, the backfill job checks the old value is unchanged since it was read. If a write raced it, the copy is skipped and the record is requeued for another pass (a "retry"). No data is ever lost, only delayed.
- Blind overwrite (unsafe) — the backfill job writes whatever it read, no questions asked. If a write raced it, the newer value gets silently clobbered by the stale one — a real conflict / data-loss event, and that record is never revisited.
Backfill duration ≈ N / backfillRate. The race window only matters while writes and backfill overlap, so raising the write rate or lowering the backfill rate widens the window and (in blind mode) raises the conflict count — try it and watch the red flashes.
Real-world relevance: this is the exact class of bug behind documented outages at companies that migrated schemas with a naive "copy everything" script instead of a version-checked backfill (GitHub, Stripe and Shopify have all written up variants of this pattern).