Configuration and environment tests: what they catch and the release incidents they cover
Configuration and environment tests (checks that config, flags, secrets and endpoints resolve correctly per environment) run at the pre-deploy and post-deploy stage.
24 published incident types list configuration and environment tests as a class that catches them; 1 of these links are cited in a public report and 23 are by design analysis.
What configuration and environment tests catch and cannot catch
Catches
- env mismatches
- flag defaults
- missing secrets
- IAM denials
- certificate expiry
Cannot catch
- logic regressions
- load
Both lists come from the site's test class definitions.
Incident types this class catches
| Incident | Stage | Limitation on that page | Basis |
|---|---|---|---|
| Encoding or timezone shift | pre-deploy | A configuration test that only checks the timezone variable on the serving layer misses a code change that drops the timezone inside a warehouse driver connection. | cited in [1] |
| Autoscaling policy misconfigured | pre-deploy | A validity check of the scaling configuration passes, because a threshold that is too high or out of date is still a valid value. | by design analysis |
| Certificate expired | post-deploy | An expiry date check on the certificate store reports the renewed certificate as valid even when the deployment reached only part of the cloud environment and some nodes still serve the old one. | by design analysis |
| Rate limit misconfigured | pre-deploy | A check of limit values per environment passes when each value is valid, so it misses the invalid limiter state that a partial flag rollout to a subset of users produced. | by design analysis |
| Schema drift between services | post-deploy | A deployed-version check confirms each service runs the version in the release manifest, but it cannot tell whether two manifest versions speak the same contract. | by design analysis |
| All sessions invalidated | pre-deploy | A configuration test that confirms the signing secret is present and unchanged misses a cookie format change made in code and a session store split between instances on different releases. | by design analysis |
| Asset cache serves old bundle | pre-deploy | A configuration check on the build output finds JavaScript URLs without a cache-busting hash, but it cannot see a stale chunk already held at a CDN edge with a success response. | by design analysis |
| Container image base change | pre-deploy | A check that the rebuilt image contains the expected user account catches the missing user, but it does not compare the image's packet filter backend with what the host kernel supports. | by design analysis |
| Dependency removed from registry | pre-deploy | A configuration check that lists image references in deployment values does not see versions pinned inside a dependency chain, such as the string padding package 0.0.3. | by design analysis |
| DNS change propagation outage | pre-deploy | A configuration check run per change on staging passed for both the certificate change and the DNS change, because the delete-before-create order appeared only when production applied the two together. | by design analysis |
| Flag enabled for wrong segment | pre-deploy | A configuration test can show that an internal-test flag resolves to every first-party application, but it does not show that the resulting client and server mismatch breaks new-user logins. | by design analysis |
| IAM policy change blocks service | pre-deploy | A configuration check runs only on changes that pass through the release pipeline, so it misses a security tool that wrote an account-level setting directly and a policy change made inside the compute provider. | by design analysis |
| OAuth callback URL not whitelisted | pre-deploy | A configuration check reads only the product's own settings, so it cannot see the redirect URIs that each customer registered in their own identity provider. | by design analysis |
| Partial rollback leaves inconsistent state | post-deploy | A version check across all eight servers finds the one that missed the new code, but it does not catch logic regressions such as the table of contents patch. | by design analysis |
| Password reset link broken | post-deploy | A check of the application URL passes when the host is right, so it misses a query string escaped into an invalid entity or a template variable that was never substituted. | by design analysis |
| Release marker missing in monitoring | pre-deploy | Configuration checks can reject the bad service record before it is pushed, but they do not catch load, and the missing index surfaced only as slow queries under production traffic. | by design analysis |
| Alerts silent after metric rename | pre-deploy | A check of alert rules against a live metrics server finds the missing failed-spans counter only after a server it reads runs the renamed collector release. | by design analysis |
| API key rotation breaks integrations | pre-deploy | A check that the current key set resolves passes before the bulk job runs, so it misses selection logic that disables new keys or ignores a retain state. | by design analysis |
| API version dropped while old app live | pre-deploy | A configuration check confirms the end-of-life date value and its rollback path, but it has no view of which installed clients still send the retired version. | by design analysis |
| Config change without restart semantics | post-deploy | A configuration check confirms the new secret is stored, but it does not show which secret version each running process holds until that process restarts. | by design analysis |
| Deep link broken after routing change | pre-deploy | A configuration test finds the redirect only if it runs on the domain migration itself, because the DNS provider move happened outside any app release. | by design analysis |
| Environment mismatch: production points to staging | pre-deploy | A check of the service's own config files passes here, because the staging reference came from the release list that the deployment API returned at promotion time. | by design analysis |
| Flag default flips on missing config | pre-deploy | A configuration test can confirm that every region has each flag and that defaults match production, but it does not measure the load the legacy HTTP path put on the internal network once defaults moved traffic onto it. | by design analysis |
| Localisation strings missing | pre-deploy | A comparison of locale files and string keys at the release gate flags the missing file or key, but not the loader rule that turned whole pt-PT documents to en-US when one of their files was missing. | by design analysis |
For cited rows the number shows how many reports on that incident page name this class.
Only on this page
Configuration and environment tests are cited most often for Encoding or timezone shift: 1 public report on that page name this class. The limitation stated there: A configuration test that only checks the timezone variable on the serving layer misses a code change that drops the timezone inside a warehouse driver connection.
Catalogue entries that name this class
From their own documentation or site, with the verification date. Alphabetical. Not a recommendation.
Tools whose documentation names these test classes
- Flipt config source verified 2026-10-21
- GrowthBook config source verified 2026-10-21
- LaunchDarkly config source verified 2026-10-21
- PostHog config source verified 2026-10-21
- Unleash config source verified 2026-10-21
Providers whose site names these test classes
No provider in the catalogue names this test class on its site.
Questions
What is Configuration and environment tests?
Configuration and environment tests: checks that config, flags, secrets and endpoints resolve correctly per environment. They run at the pre-deploy and post-deploy stage.
What does Configuration and environment tests catch?
Configuration and environment tests catch env mismatches, flag defaults, missing secrets, IAM denials and certificate expiry. 24 published incident types list this class.
What can Configuration and environment tests not catch?
Configuration and environment tests cannot catch logic regressions and load.
How is Configuration and environment tests different from other test classes?
Configuration and environment tests share the pre-deploy and post-deploy stage with contract tests, integration tests, end-to-end tests, migration tests, load and performance tests, security tests, compatibility tests, canary and progressive delivery checks, synthetic monitoring and chaos and resilience tests; they differ in what each catches.
Can Configuration and environment tests run in a production environment?
Yes. Configuration and environment tests run at the post-deploy stage, against production.
Which incidents is Configuration and environment tests used for?
alerts silent after metric rename, all sessions invalidated, aPI key rotation breaks integrations, aPI version dropped while old app live, asset cache serves old bundle, autoscaling policy misconfigured, certificate expired, config change without restart semantics, container image base change, deep link broken after routing change, dependency removed from registry, dNS change propagation outage, encoding or timezone shift, environment mismatch: production points to staging, flag default flips on missing config, flag enabled for wrong segment, iAM policy change blocks service, localisation strings missing, oAuth callback URL not whitelisted, partial rollback leaves inconsistent state, password reset link broken, rate limit misconfigured, release marker missing in monitoring and schema drift between services. 1 of 24 links are cited in a public report; the rest are by design analysis.