Thundering herd on restart: pre-release and first 30 minutes checklists

Clients reconnect simultaneously after a rolling restart and overload a dependency.

The pre-release checklist maps each item to a test class from the incident page. The first-30-minutes checklist covers detection, containment, the rollback decision and customer communication.

7 pre-release items · 8 first-30-minutes items · verified 2026-10-21

Pre-release checklist

PR-01Restart every backend pod at once in staging under peak-shaped traffic and confirm client reconnection attempts follow exponential backoff. chaos
PR-02Inject connection timeouts on one key-value store replica and confirm clients back off instead of reconnecting in a surge. chaos
PR-03Multiply the connections each process opens to the key-value store by the planned process count and confirm the total stays under the per-node connection limit. load
PR-04Run a load test with autoscaling enabled and fail when any key-value store node reaches its 65,000 connection limit as the pod count rises. load
PR-05Measure database connection count at peak traffic with the feature flag that raises client query frequency switched on, and confirm it stays below the database connection limit. load
PR-06Cap the deployment surge so each rollout step adds a fixed number of containers, and hold each step until connection errors return to baseline. canary
PR-07Compare database connection count on canary pods with baseline before promoting a deployment to all pods. canary

First 30 minutes

FM-01Check whether HTTP errors began with a burst of connection timeouts on the key-value store or with saturated database connections.
FM-02Compare the running pod count with the count before the incident and raise the autoscaling threshold if pods are multiplying.
FM-03Pause all deployments until connection counts on the saturated store return to baseline.
FM-04Add nodes or baseline capacity to the store whose connections are saturated.
FM-05Revert the feature flag change that raised client query frequency before rolling back code, because a rollback deployment adds another reconnect wave.
FM-06Reset backend pods in small batches once capacity is raised.
FM-07Post a status update that names the affected surfaces, such as the web interface, the API and job dispatch.
FM-08Tell customers with stuck jobs to cancel and retry them once dispatch delay returns to normal.

Download the checklist

Pre-release checklist (Markdown)1.6 KB · v1.0.0
First 30 minutes checklist (Markdown)1.2 KB · v1.0.0

Only in this checklist

Every test class in this checklist also appears in at least one sibling checklist in the group; the item texts differ: Restart every backend pod at once in staging under peak-shaped traffic and confirm client reconnection attempts follow exponential backoff.

Questions

What should be checked before a release to prevent Thundering herd on restart?

7 pre-release items, PR-01 to PR-07. PR-01: Restart every backend pod at once in staging under peak-shaped traffic and confirm client reconnection attempts follow exponential backoff.

Is there a rollback plan for Thundering herd on restart?

The first-30-minutes checklist has 8 items. FM-01: Check whether HTTP errors began with a burst of connection timeouts on the key-value store or with saturated database connections.

What should a team do in the first 30 minutes after Thundering herd on restart is detected?

The first-30-minutes checklist has 8 items. FM-01: Check whether HTTP errors began with a burst of connection timeouts on the key-value store or with saturated database connections.

Is a Thundering herd on restart checklist still needed if deployment is automated?

Yes. Automation runs the steps; the checklist names what the tests must cover: chaos and resilience tests, load and performance tests and canary and progressive delivery checks.

Which pipeline stages does the Thundering herd on restart checklist cover?

The pre-release items map to the pre-deploy and post-deploy stages.

In what formats can the Thundering herd on restart checklist be downloaded?

Both checklists download as Markdown files: /downloads/thundering-herd-on-restart-pre-release.md and /downloads/thundering-herd-on-restart-first-30-minutes.md, licensed CC BY 4.0.