SDK upgrade changes behaviour: symptoms, cause, and the test that catches it

An SDK upgrade changes defaults such as consent, tracking or retry behaviour.

SDK upgrade changes behaviour is documented by 3 accepted public reports from 3 organisations, 3 of them published in the last 5 years.

Likelihood: medium (3 public reports in 5 years)
Impact: medium (2 of 3 reports: availability)

3 public reports · verified 2026-10-21

Symptoms

User-visible

Cache downloads by a CI job runner from third-party object storage providers failed after the runner picked up a new storage SDK version.[1]
Backups to a third-party object storage provider failed with the error Unsupported header 'x-amz-checksum-mode' received for this API call.[2]
Clients on new SDK defaults received header errors from a compatible storage API, such as Header 'x-amz-checksum-algorithm' with value 'CRC32' not implemented.[3]

Monitoring-visible

GetObject requests to custom storage endpoints returned SignatureDoesNotMatch errors.[1]
Requests to an internal compatible object storage returned exceptions with status code 400 after the Java SDK moved to 2.30.0 or later.[4]

Root causes

stage: code The Go SDK storage module v1.73.0 changed RequestChecksumCalculation and ResponseChecksumValidation defaults from WhenRequired to WhenSupported. The new response default added X-Amz-Checksum-Mode: ENABLED as a signed header to every GetObject request, and third-party providers do not recognise that header.[1]
stage: build The backup tool picked up JavaScript SDK version 3.729 or later, and from that version the SDK sends checksum headers that the backup target did not yet support.[2]
stage: third-party The cloud vendor turned on CRC32 checksums by default for multiple object operations in its SDKs, while the storage provider's compatible API did not support CRC32 checksums.[3]

Public examples

[1]GitLab, 2026-04-29. GitLab Runner picked up AWS SDK Go v2 service/s3 v1.73.0, which changed checksum calculation and validation defaults to WhenSupported, so every GetObject request carried a new signed header and cache downloads from third-party S3-compatible providers failed with SignatureDoesNotMatch. primary Fix S3 cache broken on third-party providers by SDK checksum defaults
[2]Vates (Xen Orchestra), 2025-02-27. After Xen Orchestra picked up AWS SDK for JavaScript 3.729 or later, the SDK sent checksum headers that Backblaze did not support, so S3 backups to Backblaze failed with an unsupported header error until a merged fix restored the previous SDK behaviour. primary fix(fs): improve compatibility with backblaze
[3]Cloudflare, 2025-01-17. AWS SDK releases turned on CRC32 checksums by default for multiple object operations, and because R2 did not support CRC32 checksums, R2 clients using the new SDK defaults received header errors until Cloudflare implemented a fix on 3 February 2025. primary AWS S3 SDK compatibility inconsistencies with R2

Practitioner sources cited for symptoms

[4]Not named in the report, 2025-02-08. A developer reports that after upgrading the AWS SDK for Java 2.x S3 module to 2.30.0 or later, which enables data integrity protections by default, requests to an internal S3-compatible object storage started failing with HTTP 400 errors. practitioner When using an S3-compatible Object Storage, be cautious when upgrading SDK for Java 2.x to version 2.30.0 or later

Test classes that catch it

Test class	Stage	Limitation	Basis
Integration tests	pre-deploy	An integration suite that runs only against the SDK vendor's own storage service stays green, because the new checksum headers failed only on third-party providers configured as custom endpoints.	by design analysis
Contract tests	pre-merge	A contract on outgoing request headers flags the new x-amz-checksum-mode header only when it is checked against each third-party provider's accepted headers, not only against the vendor's API.	by design analysis
Synthetic monitoring	post-deploy	A storage provider's probe that keeps an older SDK version pinned keeps passing, because only clients on the new checksum defaults received the CRC32 header error.	by design analysis

Risk

Likelihood: medium (3 public reports in 5 years)
Impact: medium (2 of 3 reports: availability)

Counts behind the grades
Accepted public reports	3
Published in the last 5 years	3
Impact stated: degraded	1
Impact stated: availability	2

How the grades are computed

Pre-release checklist

PR-01Run the object storage integration suite against every third-party compatible provider the product supports as a custom endpoint, not only against the SDK vendor's own service. integration
PR-02When the lockfile changes the storage SDK module version, rerun the cache download and backup upload tests against each supported custom endpoint before release. integration
PR-03Confirm the client sets RequestChecksumCalculation and ResponseChecksumValidation to WhenRequired for custom non-vendor endpoints, and run one GetObject test against such an endpoint. integration
PR-04Record the signed headers each storage request sends in a contract, and fail the build when a new header such as x-amz-checksum-mode appears. contract
PR-05Keep a contract per supported third-party provider that lists the checksum headers it accepts, and check the client's recorded requests against it. contract
PR-06On the storage provider side, run a scheduled probe that uses the newest SDK release of each supported language with default configuration and alerts on any checksum header error. synthetic

First 30 minutes

FM-01Check whether the failing clients picked up a new storage SDK version, and read that version's changelog for changed checksum defaults.
FM-02Search request logs for SignatureDoesNotMatch, status 400 or unsupported header errors that name x-amz-checksum headers.
FM-03Split failures by storage endpoint to confirm that only third-party compatible providers fail.
FM-04Decide between pinning the SDK to the prior version and setting both checksum options back to WhenRequired for custom endpoints.
FM-05Publish the environment variable or client setting that restores the prior checksum behaviour, so users can unblock themselves before a fix ships.
FM-06Post a status update that names the affected SDK versions and the affected storage providers.
FM-07Rerun the cache downloads and backup jobs that failed during the incident window once the fix is in place.

Download the checklist

Pre-release checklist (Markdown)1.5 KB · v1.0.0
First 30 minutes checklist (Markdown)1.2 KB · v1.0.0

What catches this: catalogue entries

Listed because their own documentation or site names integration tests, contract tests, synthetic monitoring on the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Checkly synthetic source verified 2026-10-21
Datadog Synthetic Monitoring synthetic source verified 2026-10-21
Grafana Cloud Synthetic Monitoring synthetic source verified 2026-10-21
Grafana k6 synthetic source verified 2026-10-21
Hoverfly integration source verified 2026-10-21
Pact contract source verified 2026-10-21
Postman integration source verified 2026-10-21 synthetic source verified 2026-10-21
Schemathesis contract source verified 2026-10-21
Serenity BDD integration source verified 2026-10-21
Specmatic contract source verified 2026-10-21
Testcontainers integration source verified 2026-10-21
TestNG integration source verified 2026-10-21
Uptime Kuma synthetic source verified 2026-10-21

Providers whose site names these test classes

andagon people GmbH integration source verified 2026-10-21
Appmatics integration source verified 2026-10-21
AQA Masters integration source verified 2026-10-21
Cleverix integration source verified 2026-10-21
Crestech Software Systems integration source verified 2026-10-21
DeviQA integration source verified 2026-10-21
Full Scale contract source verified 2026-10-21
Invensis Technologies integration source verified 2026-10-21
Outpost QA integration source verified 2026-10-21
Q-Centric GmbH integration source verified 2026-10-21
QA Madness contract source verified 2026-10-21 integration source verified 2026-10-21
QA Mentor integration source verified 2026-10-21
QASource contract source verified 2026-10-21
QAwerk integration source verified 2026-10-21
Quality Island contract source verified 2026-10-21 integration source verified 2026-10-21
Soflab Technology integration source verified 2026-10-21
SQA - Software Quality Assurance S.A. integration source verified 2026-10-21
SunTec India integration source verified 2026-10-21
TestFort integration source verified 2026-10-21
Testhouse integration source verified 2026-10-21
Testing IT integration source verified 2026-10-21
TestMatick integration source verified 2026-10-21
Testscenario integration source verified 2026-10-21
Zoonou integration source verified 2026-10-21

Only on this page

Report [1], report [2] and report [3] trace SDK upgrade changes behaviour to one upstream change. The cloud vendor's SDKs turned on checksums by default for multiple object operations [3]. That change reached a CI job runner through the Go SDK at v1.73.0 [1]. It reached a backup tool through the JavaScript SDK at 3.729 [2]. It reached a storage provider's compatible API through its users' clients [3].

An integration test that runs only against the SDK vendor's service does not see this fault. The new checksum headers broke only third-party providers that implement the same storage API [1][2][3]. The fixes in reports [1] and [2] kept the upgraded SDK and restored the previous checksum behaviour, and report [1] did so only for custom non-vendor endpoints. The storage provider in report [3] added native CRC32 checksum handling and resolved the incident on 3 February 2025.

Provider API change breaks contract A third-party API changes a field or behaviour and the integration fails.
Analytics event schema broken Tracking events change shape and downstream dashboards or experiments break.
Dependency removed from registry A package or version is pulled from a registry and builds or deploys fail.
In-app purchase receipt validation fails Store receipts are rejected after a validation or sandbox endpoint change.
Third-party rate limit hit A new call pattern exceeds a provider's quota and requests are rejected.

Questions

What causes SDK upgrade changes behaviour?

In all three primary reports, the cloud vendor's SDKs turned on request or response checksums by default [3]. Third-party storage providers that implement the same API rejected the new checksum headers [1][2].

What are the symptoms of SDK upgrade changes behaviour?

GetObject requests returned SignatureDoesNotMatch errors and cache downloads failed [1]. Backups failed with an unsupported x-amz-checksum-mode header error [2].

Why did tests not catch SDK upgrade changes behaviour?

None of the reports names a test class. The change came in through a dependency version bump, v1.73.0 of the Go SDK storage module in report [1] and 3.729 of the JavaScript SDK in report [2]. A test against the SDK vendor's own service would not show the header rejection.

Which public reports document SDK upgrade changes behaviour?

Three primary reports from three organisations document it, published between 2025-01-17 [3] and 2026-04-29 [1]. One practitioner report adds status code 400 errors after a Java SDK upgrade to 2.30.0 [4].

What should be checked before a release to prevent SDK upgrade changes behaviour?

PR-01 runs the object storage integration suite against every supported third-party provider. PR-04 fails the build when a new signed header such as x-amz-checksum-mode appears in a storage request.

Should a team roll back or fix forward when SDK upgrade changes behaviour happens?

Reports [1] and [2] fixed forward by restoring the previous checksum behaviour on the upgraded SDK. The storage provider in report [3] told users they could pin the SDK to a prior version or configure the prior behaviour. FM-04 covers this decision.

Release Incident Playbook (2026). SDK upgrade changes behaviour: symptoms, cause, and the test that catches it, content version 1.0.0. https://releasefailures.com/incidents/sdk-upgrade-changes-behaviour/. Accessed 2026-10-21.