Partial rollback leaves inconsistent state: pre-release and first 30 minutes checklists

Only some services are rolled back and the system runs with incompatible versions.

The pre-release checklist maps each item to a test class from the incident page. The first-30-minutes checklist covers detection, containment, the rollback decision and customer communication.

7 pre-release items · 7 first-30-minutes items · verified 2026-10-21

Pre-release checklist

PR-01Add a unit test that renders the table of contents with the previous release's code from a cache entry written by the new release. unit
PR-02Add a version field to the cached page output format and a unit test that the reader rejects entries with an unknown version. unit
PR-03Run the automatic rollback on staging with one service failing to initialise and confirm every service returns to the previous version. chaos
PR-04Roll back on staging after the new release has written cache entries and confirm pages still render a table of contents. chaos
PR-05Uninstall the new code from all but one staging server and confirm no request reaches a code path that a repurposed flag activates. chaos
PR-06After each deploy, confirm every server in the order-routing pool reports the same code version. config
PR-07Search every build still installed on a server for code paths a flag activates before that flag is given a new meaning. config

First 30 minutes

FM-01List the version each service and server runs right now and mark every one that differs from the intended version.
FM-02If services run mixed versions and keep acting on requests, stop incoming work to the affected cluster.
FM-03Check caches and stores for entries the new version wrote before rolling any code back.
FM-04Decide between a manual downgrade of every service and a roll forward, based on whether the old version can read data the new version wrote.
FM-05Compare the times of new client error alerts with the rollback time before treating them as a separate fault.
FM-06Confirm every service and server reports the same version after the downgrade or roll forward finishes.
FM-07Post a status update that names the affected environment and the degraded features.

Download the checklist

Pre-release checklist (Markdown)1.5 KB · v1.0.0
First 30 minutes checklist (Markdown)1.1 KB · v1.0.0

Only in this checklist

PR-01 is the only item among 5 published checklists in this group that relies on unit tests: Add a unit test that renders the table of contents with the previous release's code from a cache entry written by the new release.

Questions

What should be checked before a release to prevent Partial rollback leaves inconsistent state?

7 pre-release items, PR-01 to PR-07. PR-01: Add a unit test that renders the table of contents with the previous release's code from a cache entry written by the new release.

Is there a rollback plan for Partial rollback leaves inconsistent state?

The first-30-minutes checklist has 7 items. FM-01: List the version each service and server runs right now and mark every one that differs from the intended version.

What should a team do in the first 30 minutes after Partial rollback leaves inconsistent state is detected?

The first-30-minutes checklist has 7 items. FM-01: List the version each service and server runs right now and mark every one that differs from the intended version.

Which configuration checks apply to Partial rollback leaves inconsistent state?

PR-06: After each deploy, confirm every server in the order-routing pool reports the same code version. PR-07: Search every build still installed on a server for code paths a flag activates before that flag is given a new meaning.

Is a Partial rollback leaves inconsistent state checklist still needed if deployment is automated?

Yes. Automation runs the steps; the checklist names what the tests must cover: unit tests, chaos and resilience tests and configuration and environment tests.

Which pipeline stages does the Partial rollback leaves inconsistent state checklist cover?

The pre-release items map to the pre-merge, pre-deploy and post-deploy stages.