Release Incident Playbook

Thundering herd on restart: symptoms, cause, and the test that catches it

Clients reconnect simultaneously after a rolling restart and overload a dependency.

Thundering herd on restart is documented by 2 accepted public reports from 2 organisations, 2 of them published in the last 5 years.

  • Likelihood: low (2 public reports in 5 years)
  • Impact: medium (2 of 2 reports: availability)

2 public reports · verified 2026-10-21

Symptoms

User-visible

  • API requests failed and errors rose across the web interface, the REST API and the agent API.[1]
  • Job dispatch was delayed by up to 10 minutes.[1]
  • Some jobs got stuck and needed a manual cancel and retry.[1]
  • Some users could not access their projects during system-wide failures.[2]

Monitoring-visible

  • A burst of key-value store connection timeouts targeted a single replica, together with elevated HTTP request errors.[1]
  • Several key-value store nodes hit their limit of 65,000 connections.[1]
  • The running pod count doubled within a minute and reached about five times the starting count.[1]
  • Database connections saturated after a production deployment during peak traffic.[2]
  • Client reconnection attempts added pressure on the system while database connections were saturated.[2]

Root causes

  1. stage: infra A platform migration to more, smaller application processes with fewer threads each raised total connection demand on the key-value store.[1]
  2. stage: infra Connection timeouts made clients reconnect and retry in a surge, and autoscaling added pods that each opened new connection pools, which fed the surge in a loop.[1]
  3. stage: config A feature flag change on 2025-08-19 made clients query the server more often.[2]
  4. stage: release-process A production deployment during peak traffic on 2025-08-20 set off client reconnection attempts that, with inefficient database connection management, saturated database connections.[2]

Public examples

  1. [1]Buildkite, 2026-09-11. After a migration to more, smaller application processes raised Redis connection demand, Redis connection timeouts made clients reconnect and retry in a surge, autoscaling added pods that each opened new connection pools, and a later routine deployment added containers that caused a second period of errors. primary Buildkite service disruption
  2. [2]Hex, 2025-08-26. A feature flag change made clients query the server more often, and a production deployment during peak traffic the next day triggered client reconnection attempts that saturated database connections until capacity was raised, the flag change was reverted and backend pods were reset. primary Some users are experiencing issues accessing Hex projects

Test classes that catch it

Test classStageLimitationBasis
Chaos and resilience testspre-deployA restart drill run at off-peak load misses the herd in report [2], where the deployment landed during peak traffic a day after a flag raised client query frequency.by design analysis
Load and performance testspre-deployA load test that holds the pod count fixed misses the autoscaling in report [1], where pods doubled within a minute and each new pod opened its own connection pool.by design analysis
Canary and progressive delivery checkspost-deployA canary that replaces a few pods sends only a small reconnect wave, so the database connection saturation in report [2] appears only at full deployment scale.by design analysis

Risk

  • Likelihood: low (2 public reports in 5 years)
  • Impact: medium (2 of 2 reports: availability)
Counts behind the grades
Accepted public reports2
Published in the last 5 years2
Impact stated: availability2

How the grades are computed

Pre-release checklist

  • PR-01
  • PR-02
  • PR-03
  • PR-04
  • PR-05
  • PR-06
  • PR-07

First 30 minutes

  • FM-01
  • FM-02
  • FM-03
  • FM-04
  • FM-05
  • FM-06
  • FM-07
  • FM-08

Download the checklist

What catches this: catalogue entries

Listed because their own documentation or site names chaos and resilience tests, load and performance tests, canary and progressive delivery checks on the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Providers whose site names these test classes

Only on this page

Neither report [1] nor report [2] names a code change as the cause of thundering herd on restart. Report [1] names infra: a platform migration to more, smaller processes raised connection demand, and autoscaling then multiplied pods that each opened new connection pools. Report [2] names release process: a production deployment during peak traffic set off the reconnect wave.

In report [1], a later routine deployment added containers and caused a second period of errors. Its remediation reduced the maximum deployment surge and raised the autoscaling scale-up stabilization window. In report [2], the follow-up avoids deployments during peak traffic and plans exponential backoff for client reconnection during releases. Both fixes include release and scaling settings, and report [2] also optimised database connection pooling and reuse.

Questions

What causes thundering herd on restart?

In one report, smaller processes raised connection demand and autoscaling added pods that each opened new connection pools [1]. In the other, a deployment during peak traffic set off client reconnection attempts that saturated database connections [2].

What are the symptoms of thundering herd on restart?

Users see failed API requests, job dispatch delayed by up to 10 minutes [1] or projects they cannot access [2]. Monitoring shows connection timeouts, nodes at a 65,000 connection limit [1] and saturated database connections [2].

Which test class catches thundering herd on restart, and what does it miss?

Chaos and resilience tests, load and performance tests and canary checks are listed, all with the by-design label. A load test with a fixed pod count misses autoscaling that doubled pods within a minute [1].

How long did it take to detect thundering herd on restart in the public reports?

Neither report gives a time to detect in minutes. Report [1] records elevated HTTP request errors at 18:43:10 UTC together with a burst of connection timeouts.

What should be checked before a release to prevent thundering herd on restart?

PR-01 restarts every backend pod at once in staging under peak-shaped traffic. PR-03 checks that connections per process times process count stay under the per-node limit.

Should a team roll back or fix forward when thundering herd on restart happens?

Report [2] raised capacity, reverted the flag change and reset backend pods. Report [1] added store nodes for 33 percent more connection capacity and paused deployments, since a deployment had caused a second period of errors.

Release Incident Playbook (2026). Thundering herd on restart: symptoms, cause, and the test that catches it, content version 1.0.0. https://releasefailures.com/incidents/thundering-herd-on-restart/. Accessed 2026-10-21.