Release Incident Playbook

Flag default flips on missing config: symptoms, cause, and the test that catches it

A missing or unreadable flag config falls back to the wrong default.

Flag default flips on missing config is documented by 2 accepted public reports from 2 organisations, 2 of them published in the last 5 years.

  • Likelihood: low (2 public reports in 5 years)
  • Impact: low (2 of 2 reports: degraded)

2 public reports · verified 2026-10-21

Symptoms

User-visible

  • Function execution was degraded for about 60 minutes.[1]
  • Run scheduling was degraded for about 125 minutes.[1]
  • A subset of workflow steps failed, and some outbound fetch requests from steps returned empty responses.[1]
  • EU customers found the chart formula feature in event segmentation incorrectly gated.[2]

Monitoring-visible

  • Feature flag initialization errors appeared as newly started containers failed to fetch flag values.[1]
  • DNS query volume rose roughly tenfold and DNS lookup latency spiked.[1]
  • Connection setup times rose sharply and traffic on a key internal path stalled.[1]

Root causes

  1. stage: third-party Connectivity to the external feature flag service degraded, so services could not fetch flag values at startup or through streaming updates.[1]
  2. stage: config Containers started by the new release fell back to default flag values. The defaults moved a subset of internal traffic from the RPC path to a legacy HTTP path and misconfigured another subsystem.[1]
  3. stage: config An update gated the chart formula feature, and the EU region had no feature flag for that update. EU customers therefore received the gated state.[2]

Public examples

  1. [1]Inngest, 2026-07-23. During an Inngest release on 10 July 2026, newly started containers could not fetch feature flag values from a third-party flag service and fell back to default configuration, which moved internal traffic onto a legacy HTTP path, overloaded the internal network and degraded function execution and run scheduling until the release was rolled back. primary Root Cause Analysis: July 2026 Service Incidents
  2. [2]Amplitude, 2023-07-28. An Amplitude update gated the chart formula feature, and because the EU feature flag for it was missing, EU customers fell back to the gated state for about three days. primary EU Event Segmentation Chart Formulas gated unintentionally

Test classes that catch it

Test classStageLimitationBasis
Configuration and environment testspre-deployA configuration test can confirm that every region has each flag and that defaults match production, but it does not measure the load the legacy HTTP path put on the internal network once defaults moved traffic onto it.by design analysis
Chaos and resilience testspre-deployA chaos test that cuts the flag service exercises the startup fallback, but it does not find a flag that was never created for the EU region, because that region had no dependency failure to inject.by design analysis
Canary and progressive delivery checkspost-deployA canary that receives no EU traffic does not see the gated chart formula feature, and a gated feature returns no error for the canary metrics to compare.by design analysis

Risk

  • Likelihood: low (2 public reports in 5 years)
  • Impact: low (2 of 2 reports: degraded)
Counts behind the grades
Accepted public reports2
Published in the last 5 years2
Impact stated: degraded2

How the grades are computed

Pre-release checklist

  • PR-01
  • PR-02
  • PR-03
  • PR-04
  • PR-05
  • PR-06
  • PR-07
  • PR-08

First 30 minutes

  • FM-01
  • FM-02
  • FM-03
  • FM-04
  • FM-05
  • FM-06
  • FM-07
  • FM-08

Download the checklist

What catches this: catalogue entries

Listed because their own documentation or site names configuration and environment tests, chaos and resilience tests, canary and progressive delivery checks on the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Providers whose site names these test classes

Only on this page

Flag default flips on missing config in report [1] needed a dependency outage and a release at the same time. Only containers started by the new release fell back to default flag values, because connectivity to the external flag service had degraded [1]. The defaults moved part of internal traffic onto a legacy HTTP path, and DNS query volume rose roughly tenfold [1]. A separate code regression in the same release had started customer impact at 16:10 UTC, and flag initialization errors were observed at 16:19 UTC [1].

Report [2] shows the regional form of the fault, with no outage involved. The EU region had no flag for an update that gated a chart formula feature, so EU customers received the gated state for about three days [2]. A follow-up in report [1] switched a subset of feature flags to static values.

Questions

What causes flag default flips on missing config?

In report [1], new containers could not reach the external flag service and fell back to default flag values. In report [2], the EU region had no flag for an update that gated a chart formula feature.

What are the symptoms of flag default flips on missing config?

Function execution was degraded for about 60 minutes and run scheduling for about 125 minutes [1]. EU customers found a chart formula feature incorrectly gated for about three days [2].

Which test class catches flag default flips on missing config, and what does it miss?

Configuration tests, chaos and resilience tests and canary checks are listed, all with the by-design label. A chaos test that cuts the flag service does not find a flag that was never created for the EU region.

Which public reports document flag default flips on missing config?

Two primary reports from two organisations document it. They were published on 2023-07-28 [2] and 2026-07-23 [1].

How long did it take to detect flag default flips on missing config in the public reports?

Neither report states a time to detect. In report [1], customer impact began at 16:10 UTC and flag initialization errors were observed at 16:19 UTC.

Should a team roll back or fix forward when flag default flips on missing config happens?

Report [1] rolled the release back and later switched a subset of flags to static values. Report [2] states that no data was lost and no customer action was required.

Release Incident Playbook (2026). Flag default flips on missing config: symptoms, cause, and the test that catches it, content version 1.0.0. https://releasefailures.com/incidents/flag-default-flips-on-missing-config/. Accessed 2026-10-21.