Irreversible migration fails midway: pre-release and first 30 minutes checklists

A schema migration stops halfway and cannot be rolled back, leaving tables in a mixed state.

The pre-release checklist maps each item to a test class from the incident page. The first-30-minutes checklist covers detection, containment, the rollback decision and customer communication.

7 pre-release items · 7 first-30-minutes items · verified 2026-10-21

Pre-release checklist

PR-01Run each migration against a production-sized database copy and list every step that executes outside a transaction. migration
PR-02Run the down migration for every step on the copy and mark any step without a working down path as irreversible in the pull request. migration
PR-03Split a migration that adds a unique column into separate releases for adding the column, backfilling it and adding the constraint. migration
PR-04Count rows in the source tables and the new tables after the data move step and confirm the counts match before the release proceeds. migration
PR-05Write to the table under test at production rate while the backfill step runs on the copy, then confirm the unique constraint applies. load
PR-06Kill the migration process midway on the copy and confirm the schema either returns to its previous version or resumes from the failed step. chaos
PR-07Confirm the deploy tool can roll back a partially applied migration in the staging environment. chaos

First 30 minutes

FM-01Query the migration history table to see which steps of the migration applied and which did not.
FM-02Check the database's active statements before interrupting any migration that reports a timeout.
FM-03Freeze the deployment pipeline so no further migration runs against the mixed schema.
FM-04Take a database snapshot before re-running or reverting the partially applied migration steps.
FM-05Decide between rolling forward with a data fix and a matching code version, or restoring from backup, based on which tables hold mixed data.
FM-06Deploy the application version that matches the schema as it now stands.
FM-07Post a status update that names the unavailable data or features and states whether any data was lost.

Download the checklist

Pre-release checklist (Markdown)1.5 KB · v1.0.0
First 30 minutes checklist (Markdown)1.1 KB · v1.0.0

Only in this checklist

PR-06 is the only item among 7 published checklists in this group that relies on chaos and resilience tests: Kill the migration process midway on the copy and confirm the schema either returns to its previous version or resumes from the failed step.

Questions

What should be checked before a release to prevent Irreversible migration fails midway?

7 pre-release items, PR-01 to PR-07. PR-01: Run each migration against a production-sized database copy and list every step that executes outside a transaction.

Is there a rollback plan for Irreversible migration fails midway?

The first-30-minutes checklist has 7 items. FM-01: Query the migration history table to see which steps of the migration applied and which did not.

What should a team do in the first 30 minutes after Irreversible migration fails midway is detected?

The first-30-minutes checklist has 7 items. FM-01: Query the migration history table to see which steps of the migration applied and which did not.

Is a Irreversible migration fails midway checklist still needed if deployment is automated?

Yes. Automation runs the steps; the checklist names what the tests must cover: migration tests, load and performance tests and chaos and resilience tests.

Which pipeline stages does the Irreversible migration fails midway checklist cover?

The pre-release items map to the pre-deploy and post-deploy stages.

In what formats can the Irreversible migration fails midway checklist be downloaded?

Both checklists download as Markdown files: /downloads/irreversible-migration-fails-midway-pre-release.md and /downloads/irreversible-migration-fails-midway-first-30-minutes.md, licensed CC BY 4.0.