Database Migration Rollback Plan: A Worked Compatibility Guide

Database Migration Rollback Plan: A Worked Compatibility Guide

By The Incident Challenge Team | 2026-11-08T16:21:16Z

A database migration rollback plan should name the compatible application version for each migration phase, explain how writes remain consistent, and identify the point where returning to the original schema requires a separate recovery operation.

Build that plan before the migration. A useful review includes a compatibility matrix, a rehearsed transition back, and checks for data written while versions overlap.

Amazon's rollback-safety article explains why both directions of a deployment need compatible protocols and data formats. It describes preparing readers before activating new behavior. The example below applies that general idea to an original, fictional field rename.

Start with an explicit data contract

Suppose a job record contains an integer field named timeout_ms. The team wants to rename it to request_timeout_ms, preserving the same meaning and units.

The example has one authoritative record store. Where two fields are maintained, the design requires an atomic per-record update or another mechanism that establishes the same consistency property. Storage-specific transaction, locking, and retry behavior must be verified for the actual implementation.

Define four versions of the application behavior:

Version Reads Writes Purpose
Legacy timeout_ms Old field only Original behavior
Bridge Old field Both fields consistently Prepare compatible records and writers
Activated New field, with the agreed fallback Both fields consistently Use the renamed field
Contracted New field New field only Remove the old contract after the recovery window

The names describe behavior, not product releases. Other consumers, background jobs, exports, and repair tools must be included if they read or write the record.

Phase 1: establish the bridge everywhere it is needed

Deploy the bridge behavior before depending on the new field. During this phase, readers still use the old field. Confirm that every active writer that can update these records maintains both values consistently.

A partially updated fleet cannot establish that condition. An old worker might still write only timeout_ms, leaving an existing request_timeout_ms stale.

Track the identity and revision of each relevant writer. Check scheduled and infrequent jobs as well as request-serving instances. Give any missing writer an explicit owner and keep activation blocked until the compatibility requirement is satisfied.

Phase 2: backfill with a concurrency rule

A backfill can read a record, pause, and later overwrite a value changed by an application writer. Decide how the migration prevents that stale write.

For this example, a backfill could use a record revision as a precondition and re-read after a conflict. Another implementation might perform the copy atomically within the database. The plan needs to show which guarantee the chosen mechanism provides.

Verify the result by checking records across relevant creation and update paths. A count of scanned records establishes progress; it does not establish that the resulting values are current or equal.

Phase 3: activate new reads with a tested route back

After bridge writers and the backfill are verified, introduce the activated behavior. Specify how absent values, invalid values, and conflicting fields are handled. Record which field is authoritative during the transition.

The immediate recovery target is the bridge behavior: it reads the old field, which activated writers still maintain.

A direct return to legacy-only writers while activated readers remain can create stale new-field values. The recovery sequence therefore returns readers to bridge behavior before reintroducing legacy-only writers. This ordering is part of the protocol, not an assumption hidden inside a deploy command.

Verify the transitional combinations in a suitable test environment. If the application cannot maintain the required invariant, change the design or use a different migration strategy before proceeding.

Phase 4: mark contraction as a separate decision

Removing the old field or stopping its maintenance changes the recovery options. After contraction, an old application version may no longer have the data it expects.

Require a separate review for that step. Record the observation window, remaining consumers, and the recovery plan after the old contract is removed. Possible responses include a tested repair that recreates compatible data or a restore with explicit reconciliation of subsequent writes. Their suitability depends on the system.

A backup's existence alone does not establish either response. GitLab's 2017 recovery incident is a useful companion discussion about validating recovery assumptions.

Copy the rollback planning record

Migration and business purpose:
Authoritative store and fields:
Reader/writer inventory and owners:
Compatibility behavior for each phase:
Atomicity or concurrency mechanism:
Backfill conflict and retry behavior:
Data invariants and verification queries:
Immediate rollback target for each phase:
Order of operations during recovery:
Condition that pauses or stops the migration:
Contraction decision and observation evidence:
Recovery plan after contraction:
Customer operation and data checks after recovery:

Have another engineer explain the recovery sequence using only this record. Missing assumptions become review findings before the migration becomes an incident.

Rehearse a write during the rollback

Use synthetic job records. Begin with bridge and activated instances, update a record, and recover toward the bridge. Introduce a concurrent write during the transition and check both fields against the intended value.

Repeat a different exercise after contraction. Ask what additional work is required before the legacy application can safely read the records again. Keep those two recovery problems separate in the review.

Use the canary deployment checklist to define rollout evidence and the incident runbook template to document the sequence. For prepared investigation practice, try a past Incident Challenge.