Release Incident Playbook

End-to-end tests: what they catch and the release incidents they cover

End-to-end tests (user flows through the real UI against a deployed environment) run at the pre-deploy and post-deploy stage.

17 published incident types list end-to-end tests as a class that catches them; 1 of these links are cited in a public report and 16 are by design analysis.

stage: pre-deploy, post-deploy · 17 published incident types · 1 cited

What end-to-end tests catch and cannot catch

Catches

  • broken critical paths
  • checkout failures per segment
  • login flow breaks
  • deep links

Cannot catch

  • load behaviour
  • rare data states
  • timing-dependent faults

Both lists come from the site's test class definitions.

Incident types this class catches

IncidentStageLimitation on that pageBasis
CORS policy blocks frontendpre-deployAn end-to-end login that runs only through the default subdomain passes, because the CORS error hit only workspaces reached through custom domains.cited in [1]
App store rollback not possiblepre-deployEnd-to-end runs do not reliably reproduce timing-dependent faults, and the split payment failure in app version 26.22 came from a race condition.by design analysis
SSO login brokenpre-deployAn end-to-end SSO test passes against the one or two identity provider setups it is wired to, while the reported failures hit only specific customer configurations, certificates or redirect hosts.by design analysis
All sessions invalidatedpre-deployAn end-to-end login test with a fresh session passes, because the fault hits only sessions created before the deploy and shows only while old and new instances serve the same user.by design analysis
Asset cache serves old bundlepre-deployAn end-to-end run that starts with an empty browser cache never holds the previous remote entry file, so it passes while returning browsers request assets that were removed.by design analysis
Flag enabled for wrong segmentpre-deployAn end-to-end run before the rollout passes, because the version rule switched the flag on later, when the weekly automatic upgrade changed the version it evaluated.by design analysis
OAuth callback URL not whitelistedpre-deployA login test against the team's own identity provider setup or a freshly created test setup passed in the SAML case, so an end-to-end suite misses the mismatch unless it replays a redirect_uri that an existing customer registered before the change.by design analysis
Password reset link brokenpre-deployAn end-to-end run in staging builds the reset link from the staging application URL, so it passes while production still points reset links at the pre-migration domain.by design analysis
Alerts silent after metric renamepre-deployAn end-to-end run in a healthy test deployment drops no spans, so the dropped-spans alert stays quiet whether or not its v1 metric still exists.by design analysis
Checkout blocked for segmentpre-deployAn end-to-end checkout run passes on release 11.2.0 unless its fixtures include an Argentine address with a four-digit postcode; an API-level test passes even then, because the server-side check and the store API accepted the value.by design analysis
Critical path broken in one browserpre-deployAn end-to-end test of the Files page run in any browser except the affected one passes, because the other browsers stayed fully functional.by design analysis
CSP blocks required scriptpre-deployA single end-to-end page load can pass, because the nonce attribute was dropped only on loads where the random nonce contained a + or / character.by design analysis
Deep link broken after routing changepre-deployAn end-to-end test on a device that had already validated the site association still opens the app, because the failure hit only users who installed after the migration or whose iOS rechecked the association file.by design analysis
Email or SMS provider template brokenpre-deployAn end-to-end password reset test that stops at the confirmation screen passes, because the unreplaced variable sat in the email body the user received.by design analysis
Form submission silently failspre-deployAn end-to-end test that asserts the thank-you page passes here, because visitors still got that page, so the test catches the fault only when it reads the stored submission.by design analysis
Localisation strings missingpre-deployAn end-to-end run in the default en-US locale passes, because the fallback text is correct English and the defect shows only with the language set to Russian or pt-PT.by design analysis
Push notification token resetpre-deployAn end-to-end test on a fresh install enrols a device and approves a login, but it misses the token change that happened only when the app was updated on some Android devices.by design analysis

For cited rows the number shows how many reports on that incident page name this class.

Only on this page

End-to-end tests are cited most often for CORS policy blocks frontend: 1 public report on that page name this class. The limitation stated there: An end-to-end login that runs only through the default subdomain passes, because the CORS error hit only workspaces reached through custom domains.

Catalogue entries that name this class

From their own documentation or site, with the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Providers whose site names these test classes

Questions

What is End-to-end tests?

End-to-end tests: user flows through the real UI against a deployed environment. They run at the pre-deploy and post-deploy stage.

What does End-to-end tests catch?

End-to-end tests catch broken critical paths, checkout failures per segment, login flow breaks and deep links. 17 published incident types list this class.

What can End-to-end tests not catch?

End-to-end tests cannot catch load behaviour, rare data states and timing-dependent faults.

How is End-to-end tests different from other test classes?

End-to-end tests share the pre-deploy and post-deploy stage with contract tests, integration tests, migration tests, load and performance tests, security tests, configuration and environment tests, compatibility tests, canary and progressive delivery checks, synthetic monitoring and chaos and resilience tests; they differ in what each catches.

Can End-to-end tests run in a production environment?

Yes. End-to-end tests run at the post-deploy stage, against production.

Which incidents is End-to-end tests used for?

alerts silent after metric rename, all sessions invalidated, app store rollback not possible, asset cache serves old bundle, checkout blocked for segment, cORS policy blocks frontend, critical path broken in one browser, cSP blocks required script, deep link broken after routing change, email or SMS provider template broken, flag enabled for wrong segment, form submission silently fails, localisation strings missing, oAuth callback URL not whitelisted, password reset link broken, push notification token reset and sSO login broken. 1 of 17 links are cited in a public report; the rest are by design analysis.