Release Incident Playbook

Schema drift between services: symptoms, cause, and the test that catches it

Two services expect different versions of a shared schema after one is deployed without the other.

Schema drift between services is documented by 4 accepted public reports from 4 organisations, 4 of them published in the last 5 years.

  • Likelihood: medium (4 public reports in 5 years)
  • Impact: medium (2 of 4 reports: availability)

4 public reports · verified 2026-10-21

Symptoms

User-visible

  • Some customers in all regions could not load conversational agents through the instance management portal and saw 404 errors.[1]
  • A subset of in-flight parse jobs failed during a routine deployment, affecting 31 customers and 507 jobs.[2]
  • A project overview dashboard returned 400 errors for customers on one production cluster.[3]
  • API clients across all products received timeouts and 400 to 403 statuses for about 24 minutes.[4]

Monitoring-visible

  • No alert was configured for the 404 errors, and routine service verification found the mismatch.[1]
  • Alerting fired on the parse job failures within one minute.[2]
  • Customers reported the dashboard 400 errors in chat channels at 9:30 AM UTC, and the provider acknowledged them at 9:49 AM UTC.[3]

Root causes

  1. stage: release-process A portal service shipped a new client SDK before the service it calls was updated, so the SDK called routing endpoints that were not yet deployed.[1]
  2. stage: release-process A change to an internal API contract went out by rolling deployment, and old and new service versions ran side by side during the rollout window.[2]
  3. stage: release-process A fault in the deployment workflow left some internal services undeployed, so the interface versions between services fell out of sync.[4]
  4. stage: release-process A dashboard service change was incompatible with the API contract of the service it reads from, through a gap in release orchestration coordination.[3]

Public examples

  1. [1]UiPath, 2026-01-16. UiPath updated the Agent Instance Management service with a new Conversational Agents SDK while the matching Conversational Agents service update was delayed, so the SDK called routing endpoints that did not yet exist and users received 404 errors across regions. primary Multiple Regions - Agents - Issue with Conversational agents in Agent Instance Management
  2. [2]Reducto, 2026-01-16. Reducto deployed a change to an internal API contract with a rolling deployment, and during the window when services ran mismatched versions a subset of in-flight parse jobs failed until the change was reverted. primary Increased Parse Failures
  3. [3]Harness, 2023-11-28. A Harness manager service release on Prod-2 included a dashboard service change whose API contract was incompatible with the service it consumed data from, and the Project Overview dashboard returned 400 errors until the deployment was rolled back. primary Harness Overview page showing 400 Error
  4. [4]Stitch, 2023-03-13. During a routine Stitch deployment not all internal services were deployed, so the interface versions they used to communicate fell out of sync and the API returned timeouts and 400 to 403 statuses for about 24 minutes until all services were redeployed. primary Stitch is experiencing some intermittent issues on the API

Test classes that catch it

Test classStageLimitationBasis
Contract testspre-mergeContract tests that pair the new consumer with the provider from the same commit miss the rolling deployment window, where the old provider still serves the new consumer.by design analysis
Configuration and environment testspost-deployA deployed-version check confirms each service runs the version in the release manifest, but it cannot tell whether two manifest versions speak the same contract.by design analysis
Canary and progressive delivery checkspost-deployA canary error-rate gate stays silent when 404 responses between services have no alert, as in the portal report.by design analysis

Risk

  • Likelihood: medium (4 public reports in 5 years)
  • Impact: medium (2 of 4 reports: availability)
Counts behind the grades
Accepted public reports4
Published in the last 5 years4
Impact stated: availability2
Impact stated: degraded2

How the grades are computed

Pre-release checklist

  • PR-01
  • PR-02
  • PR-03
  • PR-04
  • PR-05
  • PR-06
  • PR-07

First 30 minutes

  • FM-01
  • FM-02
  • FM-03
  • FM-04
  • FM-05
  • FM-06
  • FM-07

Download the checklist

What catches this: catalogue entries

Listed because their own documentation or site names contract tests, configuration and environment tests, canary and progressive delivery checks on the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Providers whose site names these test classes

Only on this page

Schema drift between services is introduced at the release-process stage in report [1], report [2], report [3] and report [4]. None of these reports names a database migration step.

The failure sits in deployment order and completeness. Report [4] traces it to a deployment workflow fault that left some internal services undeployed. In report [2], end-to-end tests passed on the contract change, and parse jobs still failed while services briefly ran mismatched versions during a rolling deployment.

Questions

What causes schema drift between services?

All four accepted reports place the cause at the release-process stage. Services that depend on each other were deployed out of order, partly, or by rolling deployment, so they ran incompatible contract versions.

What are the symptoms of schema drift between services?

Users see 400, 403 or 404 errors, timeouts, or failed jobs in the feature that crosses the two services. One report counts 31 customers and 507 failed parse jobs.

Why did tests not catch schema drift between services?

In the parse job report, end-to-end tests passed and jobs failed only while services ran mismatched versions. None of the four reports names a test class, so every caught-by entry on this page carries the by-design label.

Which public reports document schema drift between services?

Four primary reports are accepted, published between March 2023 and January 2026. All four name the release process as the stage where the mismatch began.

Should a team roll back or fix forward when schema drift between services happens?

Two reports rolled back, one redeployed all services, and one completed the delayed service update region by region. FM-04 asks for the rollback or the deploy, whichever restores matching versions faster.

Release Incident Playbook (2026). Schema drift between services: symptoms, cause, and the test that catches it, content version 1.0.0. https://releasefailures.com/incidents/schema-drift-between-services/. Accessed 2026-10-21.