Release Incident Playbook

Error budget burned by retry storm: symptoms, cause, and the test that catches it

Client retries amplify a partial failure into a full outage.

Error budget burned by retry storm is documented by 2 accepted public reports from 2 organisations, 2 of them published in the last 5 years.

  • Likelihood: low (2 public reports in 5 years)
  • Impact: medium (2 of 2 reports: availability)

2 public reports · verified 2026-10-22

Symptoms

User-visible

  • Server-side SDK connections to the flag delivery service failed at about 99% globally, across all regions.[1]
  • Streaming flag delivery in the United States stayed unavailable until late that night.[1]
  • About 78% of flag evaluation requests in the United States region failed with HTTP 504 errors.[2]
  • Every flag type failed during the database retry storm, including read-only flags.[2]

Monitoring-visible

  • The streaming service and its load balancer became unresponsive under SDK retry traffic.[1]
  • Infrastructure could not scale out, because compute provisioning at the cloud provider was still failing.[1]
  • Feature flags service pods failed their database connections and entered crash loops.[2]
  • Service logs showed 'pool timed out while waiting for an open connection' errors.[2]
  • Retry amplification began at 17:12 UTC, after the connection timeout change deployed at 17:02 UTC.[2]

Root causes

  1. stage: config During recovery from a cloud region outage, an internal change meant to reduce web application load reverted flag delivery to a legacy routing path with cold caches. The cold path caused excessive SDK retries, which overwhelmed the streaming service and its load balancer.[1]
  2. stage: config A deployment cut the feature flags service's database connection timeout from 1 second to 300 milliseconds while the writer database was under load from person ingestion.[2]
  3. stage: code Retry logic without circuit breakers or exponential backoff multiplied the failed connection attempts and turned a writer database load issue into full service unavailability.[2]
  4. stage: release-process The rollback of the timeout change failed because of configuration issues in the deployment tool, so service returned only after manual configuration changes.[2]

Public examples

  1. [1]LaunchDarkly, 2025-10-28. During recovery from an AWS us-east-1 outage, an internal change meant to reduce load on the web application reverted flag delivery to a legacy routing path with cold caches, SDKs retried excessively, and the retry traffic overwhelmed the streaming service and its load balancer, leaving server-side SDK connections failing at about 99% globally. primary Our October 20 Service Disruption: What Happened, What We Learned and How We’re Improving
  2. [2]PostHog, 2025-09-29. A deployment that reduced the feature flags service database connection timeout from 1 second to 300 milliseconds, during elevated writer database load, led to connection failures whose retries without circuit breakers or backoff overwhelmed the writer database and made the service unavailable. primary Feature flags service outage

Test classes that catch it

Test classStageLimitationBasis
Chaos and resilience testspre-deployA drill that drops database connections against an idle staging writer misses this failure, because the 300 millisecond timeout failed only while the writer was already under ingestion load.by design analysis
Load and performance testspre-deployA load test at normal request rates does not generate SDK retries, so it misses the retry traffic that hit the streaming service once flag delivery reverted to a cold legacy path.by design analysis
Canary and progressive delivery checkspost-deployA canary window that closes before retry amplification starts passes the timeout change, and amplification began at 17:12 UTC after the 17:02 UTC deploy.by design analysis

Risk

  • Likelihood: low (2 public reports in 5 years)
  • Impact: medium (2 of 2 reports: availability)
Counts behind the grades
Accepted public reports2
Published in the last 5 years2
Impact stated: availability2

How the grades are computed

Pre-release checklist

  • PR-01
  • PR-02
  • PR-03
  • PR-04
  • PR-05
  • PR-06
  • PR-07

First 30 minutes

  • FM-01
  • FM-02
  • FM-03
  • FM-04
  • FM-05
  • FM-06
  • FM-07

Download the checklist

What catches this: catalogue entries

Listed because their own documentation or site names chaos and resilience tests, load and performance tests, canary and progressive delivery checks on the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Providers whose site names these test classes

Only on this page

Error budget burned by retry storm in report [2] took 23 minutes to detect. The connection timeout change deployed at 17:02 UTC, retry amplification began at 17:12 UTC and the incident was declared at 17:25 UTC [2]. The rollback then failed because of deployment tool configuration issues [2]. Manual configuration changes deployed at 18:15 UTC restored service at 18:46 UTC [2].

Report [1] gives no detection time. Its retry storm began with a load-reduction change made during recovery from a cloud region outage [1]. The infrastructure in report [1] could not scale out to absorb the retries, because compute provisioning was still failing. Streaming recovered by 12:05 AM on October 21 [1].

Questions

What causes error budget burned by retry storm?

In report [2], a database connection timeout cut to 300 milliseconds met retry logic without circuit breakers or backoff. In report [1], flag delivery reverted to a legacy routing path with cold caches, and SDKs retried excessively.

What are the symptoms of error budget burned by retry storm?

Server-side SDK connections failed at about 99% globally [1]. About 78% of flag evaluation requests in the United States region failed with HTTP 504 errors [2].

Why did tests not catch error budget burned by retry storm?

The shorter timeout in report [2] failed only while the writer database was under ingestion load. Retry amplification began at 17:12 UTC, after the 17:02 UTC deploy [2].

Which test class catches error budget burned by retry storm, and what does it miss?

Chaos and resilience tests, load tests and canary checks are listed, all with the by-design label. A load test at normal request rates does not generate the SDK retries that overwhelmed the streaming service [1].

How long did it take to detect error budget burned by retry storm in the public reports?

Report [2] took 23 minutes from the 17:02 UTC deploy to the 17:25 UTC incident declaration. Report [1] does not state a detection time.

Should a team roll back or fix forward when error budget burned by retry storm happens?

In report [2], the rollback failed because of deployment tool configuration issues. Manual configuration changes then restored service at 18:46 UTC [2].

Release Incident Playbook (2026). Error budget burned by retry storm: symptoms, cause, and the test that catches it, content version 1.0.0. https://releasefailures.com/incidents/error-budget-burned-by-retry-storm/. Accessed 2026-10-21.