Release Incident Playbook

Irreversible migration fails midway: symptoms, cause, and the test that catches it

A schema migration stops halfway and cannot be rolled back, leaving tables in a mixed state.

Irreversible migration fails midway is documented by 4 accepted public reports from 4 organisations, 2 of them published in the last 5 years.

  • Likelihood: low (2 public reports in 5 years)
  • Impact: medium (2 of 4 reports: availability)

4 public reports · verified 2026-10-21

Symptoms

User-visible

  • The API was unavailable to all users for 24 minutes while the database schema and the deployed code were out of sync.[1]
  • One enterprise learner was temporarily blocked from completing enrollment while the migration sat partially applied.[2]
  • Some users could not reach document metadata such as location, person, test facility and company associations.[4]
  • Customer-facing requests timed out because the interrupted index migration could not be rolled back and the old index was already dropped.[3]

Monitoring-visible

  • Alerting fired immediately when the production migration partially failed.[1]
  • The production deployment of the migration failed and blocked the deployment pipeline.[2]
  • A unique constraint failed because the new UUID field held duplicate values.[2]
  • A migration mutex raised an 'execution expired' error in a team chat channel while the statement was still running.[3]
  • Customers reported inaccessible data on the Monday after a Saturday release.[4]

Root causes

  1. stage: migration Schema changes on the production database version were non-atomic, so a UUID backfill over about 400k rows let new records arrive, the unique constraint failed, and the migration stayed partially applied.[2]
  2. stage: migration A CREATE INDEX CONCURRENTLY step ran outside a transaction, so the interrupted migration had no rollback and the old index was already gone.[3]
  3. stage: migration A release script finished the schema modifications and then failed before moving data into the new tables and fields.[4]
  4. stage: migration A migration partially failed against production and left the schema inconsistent with the deployed API code.[1]

Public examples

  1. [1]Increase, 2024-03-26. A database migration at Increase partially failed, leaving the database schema out of sync with the deployed code, and the API was unavailable for 24 minutes until a data migration and a new API server version brought code and schema back in sync. primary API unavailability
  2. [2]edX, 2023-02-27. An edX enterprise Django migration that added tables, columns and a unique UUID field failed partway on MySQL 5.7 production and was left partially applied, blocking the deployment pipeline, and stage migrations could not be rolled back from the deploy tool. primary RCA: Enterprise partially-applied licensed course enrollment migration
  3. [3]FireHydrant, 2021-02-23. A FireHydrant index migration that ran outside a transaction was interrupted by an engineer after a misleading timeout error, and because it could not be rolled back the existing index was left dropped without its replacement, degrading runbook execution and application availability. primary February 4th, 2021 Incident Retrospective
  4. [4]CEI, 2017-06-13. A CEI ProWrite Cloud release ran a database script that completed the schema changes but failed to move data into the new tables, leaving metadata inaccessible to some users until the release was rolled back and all customer databases were restored. primary June 10, 2017

Test classes that catch it

Test classStageLimitationBasis
Migration testspre-deployMigration tests check the schema change itself, not the application code paths that read the new schema, so the API outage from code and schema falling out of step can still pass.cited in [4]
Load and performance testspre-deployLoad applied before or after the backfill misses the duplicate UUID collision, because the report states load testing would have had to run while UUID generation was running.cited in [2]
Chaos and resilience testspre-deployStopping a migration on purpose shows whether it can roll back, but it does not check data correctness, so a script that completes the schema change and skips the data move still passes.by design analysis

Risk

  • Likelihood: low (2 public reports in 5 years)
  • Impact: medium (2 of 4 reports: availability)
Counts behind the grades
Accepted public reports4
Published in the last 5 years2
Impact stated: availability2
Impact stated: degraded1
Impact stated: data1

How the grades are computed

Pre-release checklist

  • PR-01
  • PR-02
  • PR-03
  • PR-04
  • PR-05
  • PR-06
  • PR-07

First 30 minutes

  • FM-01
  • FM-02
  • FM-03
  • FM-04
  • FM-05
  • FM-06
  • FM-07

Download the checklist

What catches this: catalogue entries

Listed because their own documentation or site names migration tests, load and performance tests, chaos and resilience tests on the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Providers whose site names these test classes

Only on this page

Irreversible migration fails midway can depend on writes that arrive while the migration runs. In report [2], a backfill over about 400k rows gave new records time to land, and the unique constraint failed on a database whose schema changes are non-atomic. Report [2] states that no load testing ran against the migration, and that it would have had to run while UUID generation was running.

Report [4] names enhanced testing of data migration during schema updates as its follow-up. That release failed on the data move after the schema modifications had completed [4]. The team rolled back to the previous release and restored all customer databases [4].

Questions

What causes irreversible migration fails midway?

All four accepted reports place the cause at the migration stage. The mechanisms are a non-atomic backfill that broke a unique constraint, an interrupted index build outside a transaction, a data move that failed after the schema change, and a partial failure that left code and schema out of sync.

Why did tests not catch irreversible migration fails midway?

Report [2] states that no load testing ran against the migration while UUID generation was running. Report [2] also states that the migration saw no traffic before production.

Which public reports document irreversible migration fails midway?

Four primary reports are accepted, published between June 2017 and March 2024. Two of them name a test class: one names migration testing and one names load testing.

How long did it take to detect irreversible migration fails midway in the public reports?

Report [1] states that alerting fired immediately on the migration failure. In report [2], the failed production deployment of the migration was the first sign.

Should a team roll back or fix forward when irreversible migration fails midway happens?

One report fixed forward with a data migration and a new API server version, ending a 24-minute outage. Another rolled back to the previous release and restored all customer databases. FM-05 asks for that decision based on which tables hold mixed data.

Release Incident Playbook (2026). Irreversible migration fails midway: symptoms, cause, and the test that catches it, content version 1.0.0. https://releasefailures.com/incidents/irreversible-migration-fails-midway/. Accessed 2026-10-21.