Release Incident Playbook

Load and performance tests: what they catch and the release incidents they cover

Load and performance tests (traffic replay or synthetic load with latency and resource budgets) run at the pre-deploy stage.

14 published incident types list load and performance tests as a class that catches them; 3 of these links are cited in a public report and 11 are by design analysis.

stage: pre-deploy · 14 published incident types · 3 cited

What load and performance tests catch and cannot catch

Catches

  • N+1
  • pool exhaustion
  • cache storms
  • unbounded queries
  • autoscaling gaps

Cannot catch

  • functional defects
  • security
  • data correctness

Both lists come from the site's test class definitions.

Incident types this class catches

IncidentStageLimitation on that pageBasis
Autoscaling policy misconfiguredpre-deployA pre-production load test exercises the scaling thresholds of that environment, so it can miss outdated threshold values that exist only in the live scaling state.cited in [1]
Missing index after migrationpre-deployA load test exposes the missing index only if its traffic includes the few high traffic queries that ignored the replacement index.cited in [1]
Irreversible migration fails midwaypre-deployLoad applied before or after the backfill misses the duplicate UUID collision, because the report states load testing would have had to run while UUID generation was running.cited in [1]
Connection pool exhaustedpre-deployA load test drains the pool only if its traffic takes the specific branch that leaves a connection open, and the reported leak occurred only under certain conditions.by design analysis
Rate limit misconfiguredpre-deployA load test that sends traffic only from clients outside the allowlist never exercises the bypass path that allowlisted and Git over HTTPS traffic depended on.by design analysis
Background job queue backlogpre-deployA load test that runs for minutes misses the slow periodic query in report [3], whose instances accumulated over several days before the queue held more than 46K tasks.by design analysis
Disk full from new log volumepre-deployA load test shorter than the roughly 8 hours the root partitions took to fill from 65% to 90% ends before the disk fills, so the kernel audit log growth passes.by design analysis
Long-running migration locks tablepre-deployA load test catches connection pool exhaustion only if the schema change is applied while the test traffic is writing to the same table, and it says nothing about whether the migration itself is correct.by design analysis
Memory leak in new releasepre-deployA load test shorter than the growth period misses the leak in report [2], where file server memory built up from August 14 until nodes stopped responding.by design analysis
Cache invalidation stormpre-deployA load test sets the cache miss rate itself, so it misses the functional bug in report [1] that expired every user's settings entry at the moment of deployment.by design analysis
Cold start latency regressionpre-deployA startup benchmark run with container settings showed the new managed runtime version as faster, so a performance test misses the regression unless it reproduces the low-CPU serverless cold start.by design analysis
Error budget burned by retry stormpre-deployA load test at normal request rates does not generate SDK retries, so it misses the retry traffic that hit the streaming service once flag delivery reverted to a cold legacy path.by design analysis
Third-party rate limit hitpre-deployA load test with few remote entities and low webhook volume stays under the code host limit, because the call count grew with both webhook volume and entity count.by design analysis
Thundering herd on restartpre-deployA load test that holds the pod count fixed misses the autoscaling in report [1], where pods doubled within a minute and each new pod opened its own connection pool.by design analysis

For cited rows the number shows how many reports on that incident page name this class.

Only on this page

Load and performance tests are cited most often for Autoscaling policy misconfigured: 1 public report on that page name this class. The limitation stated there: A pre-production load test exercises the scaling thresholds of that environment, so it can miss outdated threshold values that exist only in the live scaling state.

Catalogue entries that name this class

From their own documentation or site, with the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Providers whose site names these test classes

Questions

What is Load and performance tests?

Load and performance tests: traffic replay or synthetic load with latency and resource budgets. They run at the pre-deploy stage.

What does Load and performance tests catch?

Load and performance tests catch N+1, pool exhaustion, cache storms, unbounded queries and autoscaling gaps. 14 published incident types list this class.

What can Load and performance tests not catch?

Load and performance tests cannot catch functional defects, security and data correctness.

How is Load and performance tests different from other test classes?

Load and performance tests share the pre-deploy stage with contract tests, integration tests, end-to-end tests, migration tests, security tests, configuration and environment tests, compatibility tests and chaos and resilience tests; they differ in what each catches.

Can Load and performance tests run in a production environment?

Load and performance tests run at the pre-deploy stage, before production traffic.

Which incidents is Load and performance tests used for?

autoscaling policy misconfigured, background job queue backlog, cache invalidation storm, cold start latency regression, connection pool exhausted, disk full from new log volume, error budget burned by retry storm, irreversible migration fails midway, long-running migration locks table, memory leak in new release, missing index after migration, rate limit misconfigured, third-party rate limit hit and thundering herd on restart. 3 of 14 links are cited in a public report; the rest are by design analysis.