Release Incident Playbook

Rolling deploy mixed versions: symptoms, cause, and the test that catches it

Old and new versions serve traffic at once and disagree on a contract.

Rolling deploy mixed versions is documented by 2 accepted public reports from 2 organisations, 1 of them published in the last 5 years.

  • Likelihood: not published (1 public report in 5 years)
  • Impact: medium (1 of 2 reports: availability)

2 public reports · verified 2026-10-22

Symptoms

User-visible

  • Support staff using the web app in the AU, EU and United States regions were logged out during a rolling deployment.[1]
  • Support staff could not log in or keep an active session while old and new servers both served traffic.[1]
  • The AI agent, the messenger widget, the API and inbound and outbound email kept working while web sessions failed.[1]
  • Pipeline status and the merge request widget did not load in the web UI.[2]
  • Pipelines created through web canary nodes got stuck.[2]

Monitoring-visible

  • The first failed requests appeared at 14:07:45 UTC, right after the deployment began.[1]
  • Anomaly alarms on an inbox heartbeat check fired in the EU and United States regions at 14:08 UTC.[1]
  • Error rates spiked again at 14:37 UTC when the cookie signing change was redeployed.[1]
  • The error tracker recorded errors from the background worker that handles new merge requests.[2]

Root causes

  1. stage: config A framework configuration change to the session cookie signing algorithm was not backwards compatible, so old and new application servers each rejected the other's session cookies during the rolling deployment.[1]
  2. stage: release-process A new pipeline state named preparing reached the web canary fleet while the background job workers still ran the previous code, so the two versions disagreed on the pipeline state.[2]
  3. stage: release-process Pre-production tests ran against an environment fully on the new code, so the cookie incompatibility between versions never existed there.[1]
  4. stage: release-process The rollback opened a second mixed-version window, and re-enabling automated rollbacks unlocked the pipeline, which redeployed the cookie signing change.[1]

Public examples

  1. [1]Intercom, 2026-01-05. Intercom deployed a Rails framework configuration change to the session cookie signing algorithm that was not backwards compatible; during the gradual rollout requests alternated between old and new application servers, each version rejected the other's cookies, and teammates were logged out from about 14:07 to 14:23 UTC and again from 14:37 to 14:47 UTC. primary Intercom teammate web app is down - US Hosting Status (Write-up)
  2. [2]GitLab, 2021-03-04. A merge request that introduced a new pipeline state named preparing was deployed to the GitLab.com web canary fleet while Sidekiq still ran the previous code, so pipelines created through canary nodes got stuck until canary was drained. primary 2021-03-04: Pipeline status / MR widget is not loading in Web UI (#3851)

Test classes that catch it

Test classStageLimitationBasis
Integration testspre-deployAn integration environment that runs one version of the web code and the background workers misses the window in which the web canary fleet knew the preparing state and the workers did not.by design analysis
Contract testspre-mergeA contract test on field shapes passes when a new pipeline state value is added, because the field is unchanged and only the runtime value is new.by design analysis
Canary and progressive delivery checkspost-deployA canary that compares error rates on the web canary fleet misses failures that surface in background workers still running the old code, where the stuck pipelines were processed.by design analysis

Risk

  • Likelihood: not published (1 public report in 5 years)
  • Impact: medium (1 of 2 reports: availability)
Counts behind the grades
Accepted public reports2
Published in the last 5 years1
Impact stated: availability1
Impact stated: degraded1

How the grades are computed

Pre-release checklist

  • PR-01
  • PR-02
  • PR-03
  • PR-04
  • PR-05
  • PR-06
  • PR-07

First 30 minutes

  • FM-01
  • FM-02
  • FM-03
  • FM-04
  • FM-05
  • FM-06
  • FM-07

Download the checklist

What catches this: catalogue entries

Listed because their own documentation or site names integration tests, contract tests, canary and progressive delivery checks on the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Providers whose site names these test classes

Only on this page

Rolling deploy mixed versions in report [2] took 197 minutes to detect. The web canary deployment finished at 06:54 UTC, and the incident was declared at 10:11 UTC [2]. In that window, pipelines created through canary nodes got stuck, because the background workers did not have the new preparing state [2]. The canary was drained at 10:24 UTC to stop new pipelines being exposed [2].

Report [1] detected its incident within 1 minute: failed requests appeared at 14:07:45 UTC and alarms fired at 14:08 UTC. Fast detection did not end the outage in report [1], because the rollback opened a second mixed-version window. A pipeline unlock then redeployed the cookie signing change, and error rates spiked again at 14:37 UTC [1].

Questions

What causes rolling deploy mixed versions?

In report [1], a session cookie signing change was not backwards compatible, so old and new servers rejected each other's cookies. In report [2], a new pipeline state reached the web canary fleet before the background workers had the code for it.

What are the symptoms of rolling deploy mixed versions?

Web app users were logged out and could not keep a session [1]. Pipeline status did not load and pipelines created through canary nodes got stuck [2].

Why did tests not catch rolling deploy mixed versions?

Pre-production tests in report [1] ran against an environment fully on the new code. The mixed-version incompatibility did not exist there [1].

Which test class catches rolling deploy mixed versions, and what does it miss?

Integration tests, contract tests and canary checks are listed, all with the by-design label. A canary that watches only the web canary fleet misses failures in background workers on the old code [2].

How long did it take to detect rolling deploy mixed versions in the public reports?

Report [2] took 197 minutes from the end of the canary deployment to the incident declaration. Report [1] detected failed requests within 1 minute of the deployment starting.

Should a team roll back or fix forward when rolling deploy mixed versions happens?

Report [1] rolled back manually after the automated rollback was blocked, and the rollback opened a second mixed-version window. Report [2] drained the canary fleet.

Release Incident Playbook (2026). Rolling deploy mixed versions: symptoms, cause, and the test that catches it, content version 1.0.0. https://releasefailures.com/incidents/rolling-deploy-mixed-versions/. Accessed 2026-10-21.