Release Incident Playbook

Integration tests: what they catch and the release incidents they cover

Integration tests (real dependencies (database, queue, provider sandbox) exercised together) run at the pre-deploy stage.

21 published incident types list integration tests as a class that catches them; 4 of these links are cited in a public report and 17 are by design analysis.

stage: pre-deploy · 21 published incident types · 4 cited

What integration tests catch and cannot catch

Catches

  • ORM semantics
  • webhook handling
  • connection handling
  • SDK behaviour

Cannot catch

  • production data volume
  • prod-only config
  • network conditions

Both lists come from the site's test class definitions.

Incident types this class catches

IncidentStageLimitation on that pageBasis
Provider API change breaks contractpre-deployAn integration suite that runs only on the consumer's own commits stays green when the provider changes behaviour between releases, and the merge queue service found the failing tests only after customer reports.cited in [3]
Permission escalation via role changepre-deployAn integration test that covers the optimization only while it is switched on misses the disabled code path, which is the path every workspace but one ran.cited in [1]
Analytics event schema brokenpre-deployAn integration suite catches dropped properties only when it runs against every snippet version clients still use, and this suite ran only against the upcoming snippet.cited in [1]
Email or SMS provider template brokenpre-deployAn integration test run on an install upgraded from an earlier release misses the lostPasswordUrl regression, which reached fresh installs because the change was not in the migration script.cited in [1]
Connection pool exhaustedpre-deployAn integration test can assert that each job returns its connections, but at test data volume an expensive query against the primary database does not drain a pool sized for production.by design analysis
Background job queue backlogpre-deployAn integration test that only starts a worker passes in report [1], because worker processes stayed healthy while the scheduler threads for recurring work had stopped.by design analysis
Container image base changepre-deployA job that starts the new container-in-container image on a host with nf_tables support passes, because the image failed only on hosted runner infrastructure without that support.by design analysis
Dependency removed from registrypre-deployAn integration test that passed before the removal stays green until it reinstalls, because the string padding package, the older gem builds and the cluster client image were each removed after the dependent projects had shipped.by design analysis
ORM upgrade changes query semanticspre-deployAn integration test catches the changed rows only if its fixtures hold more than one organisation, a filter field missing from the schema, or ids beyond the int range, and three of the reports show the fault appearing only for those data shapes.by design analysis
SDK upgrade changes behaviourpre-deployAn integration suite that runs only against the SDK vendor's own storage service stays green, because the new checksum headers failed only on third-party providers configured as custom endpoints.by design analysis
API key rotation breaks integrationspre-deployA run of the key cleanup job against seeded test keys leaves out relying applications that cached the published key metadata and later needed restarts.by design analysis
Config change without restart semanticspre-deployAn integration run against one database node sees either the old or the new permissions, never the mixed state that a gradual rollout creates across the cluster.by design analysis
Double charge after deploypre-deployAn integration test that sends one payment request and counts gateway submissions finds the re-execution only if it reproduces the specific timing conditions that triggered it in the transaction workflow.by design analysis
Encoding or timezone shiftpre-deployAn integration test against one warehouse connection type misses the same fault on a second connection type, and it misses a timezone variable left NULL only in the changed serving layer.by design analysis
In-app purchase receipt validation failspre-deployA sandbox purchase test runs when the purchase service or plugin changes, so it does not run when a change on the store endpoint starts returning an incorrect error in production.by design analysis
Push notification token resetpre-deployAn integration test with one app per push configuration passes, because the backend bug hit only apps that shared one p8 configuration across several apps.by design analysis
Rolling deploy mixed versionspre-deployAn integration environment that runs one version of the web code and the background workers misses the window in which the web canary fleet knew the preparing state and the workers did not.by design analysis
Soft-delete flag ignoredpre-deployAn integration test that checks only the default relationship query passes, because that path and getQuery() kept the deleted_at IS NULL condition and only toBase() lost it.by design analysis
Subscription renewal skippedpre-deployAn integration test of the payment call passes when the scheduled renewal job never selects a due membership, because no payment call is made at all.by design analysis
Tax calculation regressionpre-deployAn integration test catches the empty array only if it saves a reservation item with a price override while the tax dropdown is disabled.by design analysis
Third-party rate limit hitpre-deployAn integration test can show that one webhook event clears the whole sync cache, but at test volume it does not exhaust a provider quota that only production traffic reaches.by design analysis

For cited rows the number shows how many reports on that incident page name this class.

Only on this page

Integration tests are cited most often for Provider API change breaks contract: 3 public reports on that page name this class. The limitation stated there: An integration suite that runs only on the consumer's own commits stays green when the provider changes behaviour between releases, and the merge queue service found the failing tests only after customer reports.

Catalogue entries that name this class

From their own documentation or site, with the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Providers whose site names these test classes

Questions

What is Integration tests?

Integration tests: real dependencies (database, queue, provider sandbox) exercised together. They run at the pre-deploy stage.

What does Integration tests catch?

Integration tests catch ORM semantics, webhook handling, connection handling and SDK behaviour. 21 published incident types list this class.

What can Integration tests not catch?

Integration tests cannot catch production data volume, prod-only config and network conditions.

How is Integration tests different from other test classes?

Integration tests share the pre-deploy stage with contract tests, end-to-end tests, migration tests, load and performance tests, security tests, configuration and environment tests, compatibility tests and chaos and resilience tests; they differ in what each catches.

Can Integration tests run in a production environment?

Integration tests run at the pre-deploy stage, before production traffic.

Which incidents is Integration tests used for?

analytics event schema broken, aPI key rotation breaks integrations, background job queue backlog, config change without restart semantics, connection pool exhausted, container image base change, dependency removed from registry, double charge after deploy, email or SMS provider template broken, encoding or timezone shift, in-app purchase receipt validation fails, oRM upgrade changes query semantics, permission escalation via role change, provider API change breaks contract, push notification token reset, rolling deploy mixed versions, sDK upgrade changes behaviour, soft-delete flag ignored, subscription renewal skipped, tax calculation regression and third-party rate limit hit. 4 of 21 links are cited in a public report; the rest are by design analysis.