Release Incident Playbook

Missing index after migration: symptoms, cause, and the test that catches it

A migration drops or forgets an index and query latency rises under production load.

Missing index after migration is documented by 5 accepted public reports from 5 organisations, 3 of them published in the last 5 years.

  • Likelihood: medium (3 public reports in 5 years)
  • Impact: medium (4 of 5 reports: availability)

5 public reports · verified 2026-10-21

Symptoms

User-visible

  • Build annotations stopped displaying in the web UI after an index on the annotations table was removed.[1]
  • Workflow runs could not be created or completed through the API and SDK during a primary key migration.[2]
  • API calls timed out at call start, with about two minutes of complete downtime while database resources were expanded.[3]
  • Services were down from 11.20 to 11.28, and response times stayed slightly degraded for the rest of the day.[4]
  • Automated runbook steps such as chat channel creation ran late once queued jobs slowed without their index.[5]

Monitoring-visible

  • Queries on the annotations table timed out.[1]
  • Database CPU usage reached 100%, and unhealthy pods were restarted.[3]
  • A 5XX alarm in the Workflow API triggered about 11 minutes after the migration started in the second region.[2]
  • Recreating the index overloaded the database cluster.[4]
  • Job latency and primary database reads rose after the drop, and the read increase did not trigger an alert on its own.[5]

Root causes

  1. stage: migration A migration removed an index on the annotations table on the assumption that another index replaced it, but a few high traffic queries did not use the replacement.[1]
  2. stage: migration The old index was dropped while its replacement was unusable: one concurrent compound build had silently left the new index INVALID, and another build was interrupted after the drop.[3][5]
  3. stage: migration A primary key change dropped the index on the previous key that workflow operations relied on.[2]
  4. stage: migration A fault in the migration system removed an index related to all products during a release.[4]

Public examples

  1. [1]Buildkite, 2025-11-18. Buildkite applied a database migration that removed an index on the annotations table; some high traffic queries did not use the replacement index, queries timed out and database CPU load rose until the index was recreated on all shards. primary Increased error rate affecting Web UI and APIs
  2. [2]Onfido, 2025-02-14. During a progressive schema migration that changed a primary key, Onfido inadvertently dropped an index, endpoint latency rose and Workflow API requests failed with database query timeouts until the index was re-created. primary Workflow errors and latency
  3. [3]Vapi, 2024-10-02. Vapi dropped a simple index on its largest table after a concurrent compound index build had silently failed and left that index invalid, so the table had no usable index, database CPU reached 100% and API requests timed out until the index was rebuilt. primary API is degraded
  4. [4]Hello Retail, 2021-05-06. A fault in Hello Retail's migration system removed an index related to all products during a release, and recreating the index overloaded a database cluster, which took several services down and left degraded response times for the rest of the day. primary Stability issues
  5. [5]FireHydrant, 2021-02-23. A FireHydrant migration to replace an index with a compound index dropped the existing index before the new one was built, and the missing index slowed CloudTrail ingestion jobs that then consumed background job capacity, delaying runbook steps and causing request timeouts. primary February 4th, 2021 Incident Retrospective

Test classes that catch it

Test classStageLimitationBasis
Load and performance testspre-deployA load test exposes the missing index only if its traffic includes the few high traffic queries that ignored the replacement index.cited in [2]
Migration testspre-deployA migration test that confirms the index count after a swap passes when a replacement exists, because it does not exercise the application queries that skip that replacement.by design analysis
Canary and progressive delivery checkspost-deployCanary checks watch request traffic, so they miss a dropped index that first slows background jobs while web requests still look normal.by design analysis

Risk

  • Likelihood: medium (3 public reports in 5 years)
  • Impact: medium (4 of 5 reports: availability)
Counts behind the grades
Accepted public reports5
Published in the last 5 years3
Impact stated: availability4
Impact stated: degraded1

How the grades are computed

Pre-release checklist

  • PR-01
  • PR-02
  • PR-03
  • PR-04
  • PR-05
  • PR-06
  • PR-07
  • PR-08

First 30 minutes

  • FM-01
  • FM-02
  • FM-03
  • FM-04
  • FM-05
  • FM-06
  • FM-07

Download the checklist

What catches this: catalogue entries

Listed because their own documentation or site names load and performance tests, migration tests, canary and progressive delivery checks on the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Providers whose site names these test classes

Only on this page

Missing index after migration was detected within 5 minutes in report [1] and report [3], and within 11 minutes in report [2]. Report [1] saw queries time out on the annotations table. Report [3] saw database CPU reach 100%. Report [2] saw an alarm on a surge of 5XX errors. The longest published time to detect is about 190 minutes, from a deployment at 21:10 to the first noticed impact on runbook performance at 00:20 [5].

In report [5], the missing index slowed queued jobs rather than web requests, and the rise in primary database reads did not trigger alerts on its own. The slow jobs used up shared background capacity before user-facing requests timed out [5]. Detection for missing index after migration needs alerts on job latency and read rate as well as on endpoint errors.

Questions

What causes missing index after migration?

Every one of the five reports names a migration step as the origin. Each migration removed an index that queries still needed, through a wrong replacement assumption, an INVALID concurrent build, a primary key change or a migration system fault.

What are the symptoms of missing index after migration?

Users see API timeouts, errors and slow pages on features that read the table that lost its index. In report [1], build annotations stopped displaying in the web UI. Monitoring shows query timeouts, database CPU up to 100% [3] and rising endpoint latency.

Which test class catches missing index after migration, and what does it miss?

Load tests carry a cited basis, because one report commits to load testing schema changes before production rollout and measuring P50 and P75 latency. A load test sees the missing index only if its traffic includes the queries that needed it.

Which public reports document missing index after migration?

Five primary reports are accepted, published between February 2021 and November 2025. All five name the migration stage.

How long did it take to detect missing index after migration in the public reports?

Three reports publish 5, 5 and 11 minutes. One report publishes about 190 minutes, because the missing index slowed background jobs before any alert fired.

What should be checked before a release to prevent missing index after migration?

PR-01 asks to confirm a valid replacement index exists before any DROP INDEX runs. PR-02 asks to check the index validity flag after each concurrent index build.

Release Incident Playbook (2026). Missing index after migration: symptoms, cause, and the test that catches it, content version 1.0.0. https://releasefailures.com/incidents/missing-index-after-migration/. Accessed 2026-10-21.