API key rotation breaks integrations: pre-release and first 30 minutes checklists

A key rotation invalidates partner integrations that were not notified.

The pre-release checklist maps each item to a test class from the incident page. The first-30-minutes checklist covers detection, containment, the rollback decision and customer communication.

6 pre-release items · 7 first-30-minutes items · verified 2026-10-21

Pre-release checklist

PR-01Run the key cleanup job against a test key store seeded with old keys, keys created that day and keys marked to retain, and confirm only the old keys are disabled. integration
PR-02Run the signing key removal automation in dry-run mode and confirm no key in a retain state appears on its removal list. integration
PR-03Rotate a signing key in a test environment and confirm a relying application that cached the previous metadata still validates tokens signed with a retained key. integration
PR-04Before a bulk key operation, compare the number of keys it will disable with the number of keys unused for the cleanup period and stop if they differ. config
PR-05Confirm a point-in-time backup of the key table exists and a restore has been rehearsed before the cleanup runs. config
PR-06During and after the cleanup, call the API every few minutes with a key created in the last 24 hours and alert on any authorization error. synthetic

First 30 minutes

FM-01Check whether authorization errors are limited to API key or token traffic while the UI works.
FM-02List every key cleanup, rotation or removal job that ran before the errors began, including automated jobs outside the deploy pipeline.
FM-03Stop the running key job and freeze further bulk key operations.
FM-04Decide between restoring the affected keys from a point-in-time backup and asking customers to create new keys; a restore keeps integrations working without customer action.
FM-05For a removed signing key, roll back the published key metadata and invalidate caches that hold the changed metadata.
FM-06Post a status update that names the affected key types, states that the UI still works, and gives the new-key workaround.
FM-07Tell integration owners whether they must restart services to pick up new tokens and key metadata.

Download the checklist

Pre-release checklist (Markdown)1.4 KB · v1.0.0
First 30 minutes checklist (Markdown)1.2 KB · v1.0.0

Only in this checklist

Every test class in this checklist also appears in at least one sibling checklist in the group; the item texts differ: Run the key cleanup job against a test key store seeded with old keys, keys created that day and keys marked to retain, and confirm only the old keys are disabled.

Questions

What should be checked before a release to prevent API key rotation breaks integrations?

6 pre-release items, PR-01 to PR-06. PR-01: Run the key cleanup job against a test key store seeded with old keys, keys created that day and keys marked to retain, and confirm only the old keys are disabled.

Is there a rollback plan for API key rotation breaks integrations?

The first-30-minutes checklist has 7 items. FM-01: Check whether authorization errors are limited to API key or token traffic while the UI works.

What should a team do in the first 30 minutes after API key rotation breaks integrations is detected?

The first-30-minutes checklist has 7 items. FM-01: Check whether authorization errors are limited to API key or token traffic while the UI works.

Which configuration checks apply to API key rotation breaks integrations?

PR-04: Before a bulk key operation, compare the number of keys it will disable with the number of keys unused for the cleanup period and stop if they differ. PR-05: Confirm a point-in-time backup of the key table exists and a restore has been rehearsed before the cleanup runs.

Is a API key rotation breaks integrations checklist still needed if deployment is automated?

Yes. Automation runs the steps; the checklist names what the tests must cover: integration tests, configuration and environment tests and synthetic monitoring.

Which pipeline stages does the API key rotation breaks integrations checklist cover?

The pre-release items map to the pre-deploy and post-deploy stages.