# Thundering herd on restart: pre-release checklist

Source: https://releasefailures.com/incidents/thundering-herd-on-restart/
Content version: 1.0.0
Verified on: 2026-10-21
Licence: CC BY 4.0

Definition: Clients reconnect simultaneously after a rolling restart and overload a dependency.
- [ ] PR-01 Restart every backend pod at once in staging under peak-shaped traffic and confirm client reconnection attempts follow exponential backoff. (test class: Chaos and resilience tests)
- [ ] PR-02 Inject connection timeouts on one key-value store replica and confirm clients back off instead of reconnecting in a surge. (test class: Chaos and resilience tests)
- [ ] PR-03 Multiply the connections each process opens to the key-value store by the planned process count and confirm the total stays under the per-node connection limit. (test class: Load and performance tests)
- [ ] PR-04 Run a load test with autoscaling enabled and fail when any key-value store node reaches its 65,000 connection limit as the pod count rises. (test class: Load and performance tests)
- [ ] PR-05 Measure database connection count at peak traffic with the feature flag that raises client query frequency switched on, and confirm it stays below the database connection limit. (test class: Load and performance tests)
- [ ] PR-06 Cap the deployment surge so each rollout step adds a fixed number of containers, and hold each step until connection errors return to baseline. (test class: Canary and progressive delivery checks)
- [ ] PR-07 Compare database connection count on canary pods with baseline before promoting a deployment to all pods. (test class: Canary and progressive delivery checks)