DNS change propagation outage: pre-release and first 30 minutes checklists

A DNS or load balancer change sends traffic to the wrong target or nowhere.

The pre-release checklist maps each item to a test class from the incident page. The first-30-minutes checklist covers detection, containment, the rollback decision and customer communication.

8 pre-release items · 8 first-30-minutes items · verified 2026-10-21

Pre-release checklist

PR-01Read the infrastructure plan for the production apply and confirm no DNS record or TLS certificate is destroyed before its replacement exists. config
PR-02Apply each change tested in isolation on staging to production on its own and in sequence, and confirm the production commit matches the reviewed commit. config
PR-03Diff the target hosts of the cache load balancer in the release candidate against production and confirm each backend pool points to its intended host. config
PR-04Enable DNSSEC signing on a non-production domain that has a wildcard record before signing the main domain. config
PR-05Before removing a DS record or disabling DNSSEC signing, look up the DS record TTL at the parent zone and record how long resolvers may keep it. config
PR-06Run DNS resolution checks for every public hostname, including the app subdomain, through several validating resolvers every minute during and after the DNS change. synthetic
PR-07Download one package file from the static download host on a schedule and alert on the first name resolution failure. synthetic
PR-08Apply a cache load balancer config change to one cluster first and compare its connection failure rate with untouched clusters before continuing. canary

First 30 minutes

FM-01Resolve the failing hostname through several public and validating resolvers and record which ones fail and with which error.
FM-02Compare the current DNS records and load balancer target hosts with the last known good state.
FM-03Check whether an infrastructure apply is still running, because records it removed may return when the apply finishes.
FM-04Before rolling back a DNSSEC change, check whether resolvers have cached the DS record, because removing signing then can turn partial errors into SERVFAIL.
FM-05If a load balancer change routes traffic to the wrong host, correct the target or roll back the update and freeze further changes in that area.
FM-06After routing is fixed, drain queued CI workflow runs and watch queue delay return to normal.
FM-07Post a status update naming the affected hostnames and stating that local DNS caches may extend the outage for some users.
FM-08Ask large ISPs and public resolver operators to flush cached records when a cached DNS answer keeps failing after the fix.

Download the checklist

Pre-release checklist (Markdown)1.8 KB · v1.0.0
First 30 minutes checklist (Markdown)1.3 KB · v1.0.0

Only in this checklist

Every test class in this checklist also appears in at least one sibling checklist in the group; the item texts differ: Read the infrastructure plan for the production apply and confirm no DNS record or TLS certificate is destroyed before its replacement exists.

Questions

What should be checked before a release to prevent DNS change propagation outage?

8 pre-release items, PR-01 to PR-08. PR-01: Read the infrastructure plan for the production apply and confirm no DNS record or TLS certificate is destroyed before its replacement exists.

Is there a rollback plan for DNS change propagation outage?

The first-30-minutes checklist has 8 items. FM-01: Resolve the failing hostname through several public and validating resolvers and record which ones fail and with which error.

What should a team do in the first 30 minutes after DNS change propagation outage is detected?

The first-30-minutes checklist has 8 items. FM-01: Resolve the failing hostname through several public and validating resolvers and record which ones fail and with which error.

Which configuration checks apply to DNS change propagation outage?

PR-01: Read the infrastructure plan for the production apply and confirm no DNS record or TLS certificate is destroyed before its replacement exists. PR-02: Apply each change tested in isolation on staging to production on its own and in sequence, and confirm the production commit matches the reviewed commit. PR-03: Diff the target hosts of the cache load balancer in the release candidate against production and confirm each backend pool points to its intended host. PR-04: Enable DNSSEC signing on a non-production domain that has a wildcard record before signing the main domain. PR-05: Before removing a DS record or disabling DNSSEC signing, look up the DS record TTL at the parent zone and record how long resolvers may keep it.

Is a DNS change propagation outage checklist still needed if deployment is automated?

Yes. Automation runs the steps; the checklist names what the tests must cover: configuration and environment tests, synthetic monitoring and canary and progressive delivery checks.

Which pipeline stages does the DNS change propagation outage checklist cover?

The pre-release items map to the pre-deploy and post-deploy stages.