Release Incident Playbook

Migration safety checks before a release, drawn from public incident reports

Migration tests (migrations run against a production-shaped copy with timing and rollback checks) catch failing or slow migrations, missing indexes, locks and irreversible changes. Migration tests cannot catch faults in the application code paths that read the new schema. The 10 cited reports below show where migrations failed in production and which checks their own follow-ups name.

10 cited public reports · verified 2026-10-21

What migration tests catch and what they cannot catch

Migration tests run pre-deploy.

The migration class catches failing or slow migrations.

The migration class catches missing indexes.

The migration class catches locks.

The migration class catches irreversible changes.

The migration class cannot catch faults in application code paths that read the new schema.

The data and migrations group holds 9 incident types, and 7 of them are published.

Across the 58 published incidents, the migration class appears in 4 caught-by tables: 3 cited and 1 by design analysis.

Eight tools in the current capability matrix document migration tests.

Worked example

Missing index after migration: in a status page report from 2025, a schema migration (a scripted change to database tables, columns or indexes) removed an index on the assumption that another index replaced it [1].

A small number of high traffic queries in report [1] did not use the replacement index, and the impact was detected five minutes after the migration was applied.

The follow-ups in report [1] include migration review tooling that flags index removals and phased rollouts for high-risk migrations.

In a status page report from 2025, a progressive schema migration that changed a primary key dropped an index that workflow operations relied on [2].

Report [2] names load tests of schema changes before production rollout, measuring deviations in P50 and P75 endpoint latency, as a follow-up.

In a status page report from 2024, a concurrent compound index build failed silently and left that index invalid [3].

The team in report [3] then dropped the existing simple index, so the largest table had no usable index and database CPU reached 100%.

Long-running migration locks table: in a post-mortem from 2023, a migration adding foreign key constraints triggered a table copy that held a lock long enough to exhaust database connections [4].

The service in report [4] failed to process 74.7% of requests during a 24 minute outage.

The table in report [4] had been rated low risk because it carried under 0.2% of the busiest table's traffic.

Report [4] states that pre-production testing was not an adequate substitute for the migration running on the production dataset under full traffic load.

In a status page report from 2024, a column addition waited on a long-running SELECT, and the metadata lock it took blocked every other query on the table for about 20 minutes [5].

Irreversible migration fails midway: in a post-mortem from 2023, a migration with a unique UUID field failed partway because new records were written during a backfill (a job that writes values into existing rows after a schema or logic change) over about 400k rows [6].

The migration in report [6] was reverted and split into smaller atomic migrations in discrete deployments.

In a post-mortem from 2017, a release script completed the schema changes but failed to move data into the new tables [7].

The team in report [7] rolled back to the previous release and restored all customer databases.

Rollback fails due to forward-only change: in a post-mortem from 2021, a deploy changed a field type in a way that was not backwards compatible [8].

After the revert in report [8], rows written between the two deploys could not be read by the old code.

In a post-mortem from 2020, a post-deploy data migration passed on staging and failed on production because production held records that staging lacked [9].

A simple rollback in report [9] was not possible because some data and schema changes had already landed, so the team fixed the migration code and rolled forward.

Irreversible migration fails midway: in a status page report from 2024, a migration partially failed and left the database schema out of sync with the deployed code [10].

The API in report [10] was unavailable for 24 minutes until a data migration and a new API server version brought code and schema back in sync.

Checks before a migration reaches production

Each check below traces to a cited report in the worked example.

  1. List every query that used an index before the migration removes that index, and confirm each query uses the replacement [1].
  2. Confirm a replacement index is valid before the old index is dropped [3].
  3. Run the migration against a production-sized copy while test traffic writes to the same table [4] [6].
  4. Run load tests against each schema change and compare P50 and P75 endpoint latency with the baseline [2].
  5. Set an explicit algorithm and lock mode on every schema change statement and reject statements without them [4].
  6. Run each data migration against fixtures that hold every record shape present in production [9].
  7. Split non-atomic steps into separate small migrations shipped in discrete deployments [6].

Checks while the migration runs

Report [4] detected degradation within three minutes, and report [1] detected impact within five minutes.

  1. Check for long-running queries on the target table before a schema change starts [5].
  2. Set database-side timeouts that match client-side timeouts before the migration window [5].
  3. Watch database connection usage on the target database during the migration [4].
  4. Watch query timeouts and database CPU on the target table from the moment the migration applies until both return to baseline [1] [3].

Checks before a rollback

A rollback (returning a system to the previous release) does not reverse rows that the new release already wrote [8].

  1. Confirm the previous release can read every row the new release writes before the deploy starts [8].
  2. Name the application version that matches the schema if the migration stops partway, and keep that version ready to deploy [10].
  3. Write down whether the migration can be reversed and, if not, the roll-forward fix path [9].
  4. Confirm a restorable database copy exists for every customer database the release changes [7].

Public reports cited

  1. [1]Buildkite, 2025-11-18. Buildkite applied a database migration that removed an index on the annotations table; some high traffic queries did not use the replacement index, queries timed out and database CPU load rose until the index was recreated on all shards. primary Increased error rate affecting Web UI and APIs
  2. [2]Onfido, 2025-02-14. During a progressive schema migration that changed a primary key, Onfido inadvertently dropped an index, endpoint latency rose and Workflow API requests failed with database query timeouts until the index was re-created. primary Workflow errors and latency
  3. [3]Vapi, 2024-10-02. Vapi dropped a simple index on its largest table after a concurrent compound index build had silently failed and left that index invalid, so the table had no usable index, database CPU reached 100% and API requests timed out until the index was rebuilt. primary API is degraded
  4. [4]Pulumi, 2023-10-18. Pulumi shipped a database migration modifying foreign keys on a table; the resulting table copy held a lock long enough to pile up queries and exhaust database connections, and Pulumi Cloud failed 74.7% of requests for about 24 minutes. primary Details of the Pulumi Outage on October 6, 2023
  5. [5]Honeycomb, 2024-08-07. A column addition to the teams table, applied during a deployment while a slow SELECT was running, took a metadata lock that queued all other queries on the table, including authentication, for about 20 minutes. primary Honeycomb Status - UI and API unavailable
  6. [6]edX, 2023-02-27. An edX enterprise Django migration that added tables, columns and a unique UUID field failed partway on MySQL 5.7 production and was left partially applied, blocking the deployment pipeline, and stage migrations could not be rolled back from the deploy tool. primary RCA: Enterprise partially-applied licensed course enrollment migration
  7. [7]CEI, 2017-06-13. A CEI ProWrite Cloud release ran a database script that completed the schema changes but failed to move data into the new tables, leaving metadata inaccessible to some users until the release was rolled back and all customer databases were restored. primary June 10, 2017
  8. [8]CircleCI, 2021-11-11. A deploy changed a field type in the job distribution database in a way that was not backwards compatible; after the team reverted it, the rows written between the two deploys could not be read by the old code, so distribution kept failing until a hand-built change made the distributor ignore the field. primary Incident Report: November 8, 2021 - Jobs stuck in a "not running" state
  9. [9]GitLab, 2020-09-22. A post-deploy data migration that had passed on staging failed on GitLab.com production because production held records staging lacked; the daily deploy was blocked and a simple rollback was not possible because some data and schema changes had already landed, so the team fixed the migration code and rolled forward. primary 2020-09-22: post-deployment migration failed on Production
  10. [10]Increase, 2024-03-26. A database migration at Increase partially failed, leaving the database schema out of sync with the deployed code, and the API was unavailable for 24 minutes until a data migration and a new API server version brought code and schema back in sync. primary API unavailability

Questions

What backups should exist before a migration is released?

In report [7], a data move failed after the schema changes completed, and the team rolled back the release and restored all customer databases. That report's follow-ups include side-by-side deployments of release candidates with database copies and a review of the recovery plan.

How do you decide between rolling back and fixing forward?

Check whether the previous release can read rows the new release wrote. In report [8], rows written between two deploys could not be read after the revert. In report [9], a simple rollback was not possible, so the team fixed the migration code and rolled forward.

Why did a migration pass staging and fail in production?

In report [9], production held records that staging lacked, so a code branch in the data migration never ran on staging. Report [4] states that pre-production testing was not an adequate substitute for the production dataset under full traffic load.

How many tools in the catalogue document migration tests?

Eight tools in the current capability matrix document migration tests. The What catches this block lists them alphabetically with a source URL and a verification date, and the order is not a ranking.