Container image base change: symptoms, cause, and the test that catches it

Symptoms

Root causes

1. stage: third-party The 24.0.7 release of the container-in-container image updated its base image. The new base moved the packet filter backend from the legacy backend at 1.8.9 to nf_tables at 1.8.10.[2]
2. stage: infra The hosted runner infrastructure did not support nf_tables, so the 24.0.7 image failed to start there.[2]
3. stage: third-party The upstream official JavaScript runtime image moved one of its tags from an older to a newer major release of its Linux distribution base.[3]
4. stage: build The image build automation removed the upstream build file's FROM statement by pattern matching and appended the rest to its own build files. After the base change the pattern no longer matched, and multi-stage build support let the image build with an extra FROM statement.[3]

Public examples

1. [1]Crafty Controller, 2025-01-20. Crafty Controller changed its Docker base image from ubuntu:22.04 to ubuntu:24.04 in release 4.4.5; the new image ships a default ubuntu user with UID 1000 that conflicted with the project's own non-root user, so existing containers with mounted volumes lost file access and entered a boot loop in versions 4.4.5 and 4.4.6 until 4.4.7 removed the shipped user. primary Docker Image 24.04 rebase - non-root user issue. (#521)
2. [2]GitLab, 2023-12-15. The docker 24.0.7 image release updated its alpine base, which switched iptables to nf_tables; GitLab's SaaS runner infrastructure did not support nf_tables, so docker:24.0.7-dind failed to start and CI jobs could not build images until the upstream image was fixed. primary 2023-12-15: docker:24.0.7-dind failing to start on GitLab Runner (saas)
3. [3]CircleCI, 2018-12-08. Docker moved the official Node images' carbon tag from a Debian Jessie base to a Debian Stretch base, CircleCI's image build automation then left an extra FROM statement in Node variants of its convenience images, and the rebuilt images were missing the circleci user and many failed to start in CircleCI jobs. primary Convenience Images Update: 2018.11.27 Node variants incident

Test classes that catch it

Risk

Pre-release checklist

• PR-01Start each rebuilt CI image on the same hosted runner infrastructure that customer jobs use and confirm a job inside it builds an image. integration
• PR-02Before adopting a new container-in-container image tag, run a job on the hosted runners that builds an image and confirm the inner daemon starts. integration
• PR-03Run a command inside each rebuilt image that prints the expected user account's id and fail the build when the account is missing. config
• PR-04Fail the image build when a generated build file meant for a single-stage image contains more than one FROM statement. config
• PR-05Record the base image's distribution release and packet filter backend on every build and fail when either differs from the previous build without an approved change. config
• PR-06Publish a rebuilt image to a small share of CI jobs first and compare its job start failure rate with the previous image before moving the shared tag. canary
• PR-07Point the latest tag at a new image only after the canary share has run without a rise in job errors for the agreed period. canary

What catches this: catalogue entries

Listed because their own documentation or site names integration tests, configuration and environment tests, canary and progressive delivery checks on the verification date. Alphabetical. Not a recommendation.

Related incidents

• DNS change propagation outage A DNS or load balancer change sends traffic to the wrong target or nowhere.
• Autoscaling policy misconfigured A scaling change fails to add capacity or scales to zero under load.
• Dependency removed from registry A package or version is pulled from a registry and builds or deploys fail.
• Disk full from new log volume A logging change fills disks and services stop writing.
• Rolling deploy mixed versions Old and new versions serve traffic at once and disagree on a contract.

Questions