Release Incident Playbook

Autoscaling policy misconfigured: symptoms, cause, and the test that catches it

A scaling change fails to add capacity or scales to zero under load.

Autoscaling policy misconfigured is documented by 5 accepted public reports from 5 organisations, 4 of them published in the last 5 years.

  • Likelihood: medium (4 public reports in 5 years)
  • Impact: medium (5 of 5 reports: availability)

5 public reports · verified 2026-10-22

Symptoms

User-visible

  • Requests to the public API in the EU cluster saw increased latency and timeouts from 13:02 to 13:25 GMT after a release changed the auto-scaling rule.[5]
  • The EU API cluster had an elevated error rate and a partial outage while its serving instances sat at the off-hours baseline.[5]
  • At peak, 30 percent of CI jobs failed to start and 2 percent were delayed more than 5 minutes after an autoscaling component scaled the service below its baseline.[1]
  • Cloud coding agent sessions and static site builds were also affected during the CI capacity drop.[1]
  • Customers could not log in to seven products and services for about 42 minutes after a new configuration data store failed to scale up under traffic.[4]

Monitoring-visible

  • Monitoring alerted on increased API latency at 13:11 GMT, nine minutes after the deployment started.[5]
  • The number of available serving instances dropped to minimum baseline values sized for off-hours volume.[5]
  • Workflow processing capacity in the CI service fell below its required baseline.[1]
  • A queued backlog of workflow runs built up while capacity was reduced.[1]

Root causes

  1. stage: config A release changed the auto-scaling rule to make it more reactive, but it set the scaling threshold to a high value and cut serving instances to minimum baselines sized for off-hours volume.[5]
  2. stage: config A configuration change to an internal autoscaling component carried outdated capacity threshold values, so the CI service scaled below its required baseline.[1]
  3. stage: release-process Auto-scaling changes applied immediately, with no canary or partial rollout.[5]
  4. stage: release-process Load testing of the scaling rule change was skipped because of the timing of the merge and the deployment.[5]

Public examples

  1. [1]GitHub, 2026-07-13. A configuration change to an internal GitHub autoscaling component carried outdated capacity threshold values and scaled a GitHub Actions service below its required baseline, so up to 30 percent of Actions jobs failed to start between 13:11 and 13:53 UTC until the change was rolled back. primary Actions runs are experiencing failures to start
  2. [2]HubSpot, 2025-09-26. A HubSpot deployment created auto-scaling issues that overloaded services, and some customers in North America could not use the HubSpot application between 12:17 PM and 12:28 PM EDT on September 26, 2025. primary HubSpot Application in North America was unavailable for some users
  3. [3]LiveKit, 2025-08-22. A change affecting LiveKit's autoscaling metrics caused its canary clusters to stop scaling, and about 2 percent of StartEgress requests failed with 503 errors from 15:58 to 16:18 UTC until the change was reverted. primary Temporary Egress Availability Issues
  4. [4]Atlassian, 2022-03-14. Atlassian lowered the provisioned read and write capacity values of a newly introduced DynamoDB based configuration data store whose autoscaling was misconfigured, so the store did not scale up under traffic and logins to Trello, Statuspage and other Atlassian sites failed for about 42 minutes until use of the new store was disabled. primary Login failures
  5. [5]Onfido, 2021-06-21. An Onfido API release changed the auto-scaling rule to make it more reactive, but the change reduced serving instances to their off-hours minimum baseline, causing increased latency and timeouts on api.onfido.com in the EU from 13:02 to 13:25 GMT until the release was rolled back. primary API Elevated Error Rate

Test classes that catch it

Test classStageLimitationBasis
Load and performance testspre-deployA pre-production load test exercises the scaling thresholds of that environment, so it can miss outdated threshold values that exist only in the live scaling state.cited in [5]
Configuration and environment testspre-deployA validity check of the scaling configuration passes, because a threshold that is too high or out of date is still a valid value.by design analysis
Canary and progressive delivery checkspost-deployA canary that splits request traffic does not cover a scaling rule that applies to the whole cluster the moment it ships.by design analysis

Risk

  • Likelihood: medium (4 public reports in 5 years)
  • Impact: medium (5 of 5 reports: availability)
Counts behind the grades
Accepted public reports5
Published in the last 5 years4
Impact stated: availability5

How the grades are computed

Pre-release checklist

  • PR-01
  • PR-02
  • PR-03
  • PR-04
  • PR-05
  • PR-06
  • PR-07

First 30 minutes

  • FM-01
  • FM-02
  • FM-03
  • FM-04
  • FM-05
  • FM-06
  • FM-07

Download the checklist

What catches this: catalogue entries

Listed because their own documentation or site names load and performance tests, configuration and environment tests, canary and progressive delivery checks on the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Providers whose site names these test classes

Only on this page

In report [5], monitoring alerted on increased latency at 13:11 GMT, nine minutes after the deployment started. The new auto-scaling rule had cut serving instances to minimum baselines sized for off-hours volume. The deployment was rolled back at 13:23 GMT, and the latency and timeouts ended at 13:25 GMT. Reports [1] to [4] give no detection time.

Report [5] states that end-to-end tests passed because the scaling configuration was valid. Load testing was skipped because of the timing of the merge and the deployment. The follow-up in report [5] makes scaling configuration changes subject to load tests in the pre-production environment before release.

Questions

What causes autoscaling policy misconfigured?

In report [5], a release set a high scaling threshold and cut instances to an off-hours baseline. In report [1], an autoscaling component carried outdated capacity threshold values and scaled a CI service below its baseline.

What are the symptoms of autoscaling policy misconfigured?

Users saw API latency, timeouts and an elevated error rate [5]. In report [1], 30 percent of CI jobs failed to start at peak. In report [4], logins to seven products and services failed for about 42 minutes.

Why did tests not catch autoscaling policy misconfigured?

In report [5], end-to-end tests passed because the scaling configuration was valid. Load testing was skipped because of the timing of the merge and the deployment.

Which test class catches autoscaling policy misconfigured, and what does it miss?

Load and performance tests carry the cited label from report [5]. A pre-production load test can still miss outdated threshold values that exist only in the live scaling state, as in report [1].

How long did it take to detect autoscaling policy misconfigured in the public reports?

Report [5] gives nine minutes from the start of the deployment to the latency alert at 13:11 GMT. Reports [1] to [4] give no detection time.

Should a team roll back or fix forward when autoscaling policy misconfigured happens?

Reports [1], [3] and [5] rolled back or reverted the scaling change. Report [4] disabled use of the new configuration data store instead. In report [1], full recovery followed at 13:53 UTC after the backlog drained.

Release Incident Playbook (2026). Autoscaling policy misconfigured: symptoms, cause, and the test that catches it, content version 1.0.0. https://releasefailures.com/incidents/autoscaling-policy-misconfigured/. Accessed 2026-10-21.