Container image base change: pre-release and first 30 minutes checklists

A base image update changes libraries or users and the service fails to start.

The pre-release checklist maps each item to a test class from the incident page. The first-30-minutes checklist covers detection, containment, the rollback decision and customer communication.

7 pre-release items · 7 first-30-minutes items · verified 2026-10-22

Pre-release checklist

PR-01Start each rebuilt CI image on the same hosted runner infrastructure that customer jobs use and confirm a job inside it builds an image. integration
PR-02Before adopting a new container-in-container image tag, run a job on the hosted runners that builds an image and confirm the inner daemon starts. integration
PR-03Run a command inside each rebuilt image that prints the expected user account's id and fail the build when the account is missing. config
PR-04Fail the image build when a generated build file meant for a single-stage image contains more than one FROM statement. config
PR-05Record the base image's distribution release and packet filter backend on every build and fail when either differs from the previous build without an approved change. config
PR-06Publish a rebuilt image to a small share of CI jobs first and compare its job start failure rate with the previous image before moving the shared tag. canary
PR-07Point the latest tag at a new image only after the canary share has run without a rise in job errors for the agreed period. canary

First 30 minutes

FM-01Group failing jobs by image tag and digest to confirm the failures follow one image version.
FM-02Compare the base layer of the failing image with the last working image, including distribution release, packet filter backend and user accounts.
FM-03Tell affected users to pin jobs to the last working image tag as the immediate workaround.
FM-04If the image is built in-house, roll the shared tag back to the previous image digest.
FM-05If the image comes from upstream, report the fault to its maintainers with the host details that trigger the failure.
FM-06Post a status update that names the affected image tags and the pin workaround.
FM-07Remove the pin only after the fixed upstream or rebuilt image passes a job on the hosted runners.

Download the checklist

Pre-release checklist (Markdown)1.6 KB · v1.0.0
First 30 minutes checklist (Markdown)1.1 KB · v1.0.0

Only in this checklist

Every test class in this checklist also appears in at least one sibling checklist in the group; the item texts differ: Start each rebuilt CI image on the same hosted runner infrastructure that customer jobs use and confirm a job inside it builds an image.

Questions

What should be checked before a release to prevent Container image base change?

7 pre-release items, PR-01 to PR-07. PR-01: Start each rebuilt CI image on the same hosted runner infrastructure that customer jobs use and confirm a job inside it builds an image.

Is there a rollback plan for Container image base change?

The first-30-minutes checklist has 7 items. FM-01: Group failing jobs by image tag and digest to confirm the failures follow one image version.

What should a team do in the first 30 minutes after Container image base change is detected?

The first-30-minutes checklist has 7 items. FM-01: Group failing jobs by image tag and digest to confirm the failures follow one image version.

Which configuration checks apply to Container image base change?

PR-03: Run a command inside each rebuilt image that prints the expected user account's id and fail the build when the account is missing. PR-04: Fail the image build when a generated build file meant for a single-stage image contains more than one FROM statement. PR-05: Record the base image's distribution release and packet filter backend on every build and fail when either differs from the previous build without an approved change.

Is a Container image base change checklist still needed if deployment is automated?

Yes. Automation runs the steps; the checklist names what the tests must cover: integration tests, configuration and environment tests and canary and progressive delivery checks.

Which pipeline stages does the Container image base change checklist cover?

The pre-release items map to the pre-deploy and post-deploy stages.