DNS change propagation outage: symptoms, cause, and the test that catches it

A DNS or load balancer change sends traffic to the wrong target or nowhere.

DNS change propagation outage is documented by 3 accepted public reports from 3 organisations, 3 of them published in the last 5 years.

Likelihood: medium (3 public reports in 5 years)
Impact: medium (3 of 3 reports: availability)

3 public reports · verified 2026-10-21

Symptoms

User-visible

95 percent of CI workflow runs failed to start within 5 minutes, with an average delay of 30 minutes, after a load balancer change routed internal traffic to the wrong host.[1]
10 percent of CI workflow runs failed with an infrastructure error during the same load balancer fault.[1]
Builds failed because packages could not be downloaded while the static download host had no DNS record.[2]
For some users the missing DNS record kept failing a few minutes longer because their own DNS server cached the result.[2]
Users saw ERR_NAME_NOT_RESOLVED errors in the chat clients after DNSSEC signing was enabled on the main domain.[3]

Monitoring-visible

Connection failures appeared across backend resources while the cache load balancer pointed at an incorrect host.[1]
Queuing of CI workflow runs slowed while internal traffic went to the incorrect host.[1]
The DNS record for the static download host failed to resolve for about 10 to 15 minutes.[2]
Customer network logs showed the main domain resolving while the app subdomain failed.[3]
External DNS and DNSSEC resolution probes alerted after the DNSSEC rollback, as validating resolvers returned SERVFAIL.[3]

Root causes

stage: config Cache infrastructure updates introduced incorrect configuration into the cache load balancer, which routed internal CI traffic to an incorrect host. That report names no DNS record change.[1]
stage: infra An infrastructure-as-code apply removed the existing TLS certificate and DNS records first and recreated the DNS records only after the new certificate was issued about 10 minutes later.[2]
stage: release-process Production was set to the latest commit on the main branch, so a new certificate and updated DNS records, tested separately on staging, were applied together.[2]
stage: config NSEC responses for a wildcard record led some resolvers to cache that the name had no records, and removing a DS record that resolvers had already cached left validating resolvers returning SERVFAIL until the DS record TTL expired.[3]

Public examples

[1]GitHub, 2026-03-05. Redis infrastructure updates rolled out to production introduced incorrect configuration into the Redis load balancer, routing GitHub Actions internal traffic to an incorrect host, so between 16:24 and 19:30 UTC on March 5, 2026, 95 percent of workflow runs failed to start within 5 minutes and 10 percent failed with an infrastructure error. primary Multiple services are affected, service degradation
[2]Rust Infrastructure Team, 2023-02-08. A Terraform deployment to crates.io production removed the existing TLS certificate and DNS records before issuing a new certificate, leaving static.crates.io with no DNS record for about 10 to 15 minutes until Terraform recreated the records. primary DNS Outage on 2023-01-25
[3]Slack, 2021-11-29. Slack enabled DNSSEC signing on slack.com and published the DS record; users began to see name resolution errors, and after the change was rolled back, validating resolvers that had cached the DS record returned SERVFAIL for slack.com lookups, affecting less than 1 percent of online users for about 24 hours. primary The Case of the Recursive Resolvers: What Happened During Slack's DNSSEC Rollout

Test classes that catch it

Test class	Stage	Limitation	Basis
Configuration and environment tests	pre-deploy	A configuration check run per change on staging passed for both the certificate change and the DNS change, because the delete-before-create order appeared only when production applied the two together.	by design analysis
Synthetic monitoring	post-deploy	External DNSSEC probes went green after the DS record was published, because the failure hit only resolvers that had cached a negative answer for the wildcard name.	by design analysis
Canary and progressive delivery checks	post-deploy	A canary that compares only public request errors misses the fault, because the misconfigured cache load balancer broke internal traffic that public requests never carry.	by design analysis

Risk

Likelihood: medium (3 public reports in 5 years)
Impact: medium (3 of 3 reports: availability)

Counts behind the grades
Accepted public reports	3
Published in the last 5 years	3
Impact stated: availability	3

How the grades are computed

Pre-release checklist

PR-01Read the infrastructure plan for the production apply and confirm no DNS record or TLS certificate is destroyed before its replacement exists. config
PR-02Apply each change tested in isolation on staging to production on its own and in sequence, and confirm the production commit matches the reviewed commit. config
PR-03Diff the target hosts of the cache load balancer in the release candidate against production and confirm each backend pool points to its intended host. config
PR-04Enable DNSSEC signing on a non-production domain that has a wildcard record before signing the main domain. config
PR-05Before removing a DS record or disabling DNSSEC signing, look up the DS record TTL at the parent zone and record how long resolvers may keep it. config
PR-06Run DNS resolution checks for every public hostname, including the app subdomain, through several validating resolvers every minute during and after the DNS change. synthetic
PR-07Download one package file from the static download host on a schedule and alert on the first name resolution failure. synthetic
PR-08Apply a cache load balancer config change to one cluster first and compare its connection failure rate with untouched clusters before continuing. canary

First 30 minutes

FM-01Resolve the failing hostname through several public and validating resolvers and record which ones fail and with which error.
FM-02Compare the current DNS records and load balancer target hosts with the last known good state.
FM-03Check whether an infrastructure apply is still running, because records it removed may return when the apply finishes.
FM-04Before rolling back a DNSSEC change, check whether resolvers have cached the DS record, because removing signing then can turn partial errors into SERVFAIL.
FM-05If a load balancer change routes traffic to the wrong host, correct the target or roll back the update and freeze further changes in that area.
FM-06After routing is fixed, drain queued CI workflow runs and watch queue delay return to normal.
FM-07Post a status update naming the affected hostnames and stating that local DNS caches may extend the outage for some users.
FM-08Ask large ISPs and public resolver operators to flush cached records when a cached DNS answer keeps failing after the fix.

Download the checklist

Pre-release checklist (Markdown)1.8 KB · v1.0.0
First 30 minutes checklist (Markdown)1.3 KB · v1.0.0

What catches this: catalogue entries

Listed because their own documentation or site names configuration and environment tests, synthetic monitoring, canary and progressive delivery checks on the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Argo Rollouts canary source verified 2026-10-21
Checkly synthetic source verified 2026-10-21
Datadog Synthetic Monitoring synthetic source verified 2026-10-21
Flagger canary source verified 2026-10-21
Flipt config source verified 2026-10-21
Grafana Cloud Synthetic Monitoring synthetic source verified 2026-10-21
Grafana k6 synthetic source verified 2026-10-21
GrowthBook config source verified 2026-10-21 canary source verified 2026-10-21
Kayenta canary source verified 2026-10-21
LaunchDarkly config source verified 2026-10-21 canary source verified 2026-10-21
PostHog config source verified 2026-10-21
Postman synthetic source verified 2026-10-21
Spinnaker canary source verified 2026-10-21
Unleash config source verified 2026-10-21 canary source verified 2026-10-21
Uptime Kuma synthetic source verified 2026-10-21

Providers whose site names these test classes

No provider in the catalogue names this test class on its site.

Only on this page

In report [3], external DNSSEC probes went green after the DS record was published at the registrar, while users saw ERR_NAME_NOT_RESOLVED errors. Customer network logs showed the main domain resolving while the app subdomain failed. NSEC responses for the wildcard record had led some resolvers to cache that the name had no records.

The rollback in report [3] made the DNS change propagation outage worse for one group of users. Validating resolvers that had cached the DS record returned SERVFAIL for main domain lookups once signing was disabled. The impact lasted about 24 hours for less than 1 percent of online users, until the cached DS record expired. Report [3] calls it an oversight not to test a domain with a wildcard record first.

Autoscaling policy misconfigured A scaling change fails to add capacity or scales to zero under load.
IAM policy change blocks service A permission change removes a service's access to a queue, bucket or secret.
Rolling deploy mixed versions Old and new versions serve traffic at once and disagree on a contract.
Certificate expired A TLS certificate or pinned certificate expires and connections fail.
Container image base change A base image update changes libraries or users and the service fails to start.

Questions

What causes DNS change propagation outage?

Report [1] traces it to incorrect load balancer configuration that routed internal traffic to the wrong host, and it names no DNS record change. Report [2] traces it to an apply that deleted DNS records before recreating them. Report [3] traces it to DNSSEC signing on a domain with a wildcard record.

What are the symptoms of DNS change propagation outage?

Users saw ERR_NAME_NOT_RESOLVED errors [3] or builds failing to download packages [2]. In report [1], 95 percent of CI workflow runs failed to start within 5 minutes.

Which test class catches DNS change propagation outage, and what does it miss?

Configuration and environment tests, synthetic monitoring and canary checks carry the by-design label, because none of the three reports names a test class. Synthetic DNSSEC probes went green in report [3] while some resolvers still failed.

Which public reports document DNS change propagation outage?

Three primary reports are on this page, published in 2021, 2023 and 2026. They cover a load balancer misroute [1], a delete-before-create DNS apply [2] and a DNSSEC rollout [3].

What should be checked before a release to prevent DNS change propagation outage?

Item PR-01 confirms the production plan destroys no DNS record or certificate before its replacement exists. Item PR-04 tests DNSSEC signing on a non-production domain with a wildcard record first.

Should a team roll back or fix forward when DNS change propagation outage happens?

In report [1], the load balancer was corrected and the contributing updates were rolled back. In report [3], the DNSSEC rollback caused SERVFAIL for validating resolvers that had cached the DS record.

Release Incident Playbook (2026). DNS change propagation outage: symptoms, cause, and the test that catches it, content version 1.0.0. https://releasefailures.com/incidents/dns-change-propagation-outage/. Accessed 2026-10-21.