Memory leak in new release: pre-release and first 30 minutes checklists

Memory grows per request or per screen until processes restart or the app is killed.

The pre-release checklist maps each item to a test class from the incident page. The first-30-minutes checklist covers detection, containment, the rollback decision and customer communication.

6 pre-release items · 7 first-30-minutes items · verified 2026-10-21

Pre-release checklist

PR-01Confirm the canary receives at least the per-pod request rate of production peak before an authorization service build graduates. canary
PR-02Compare canary pod memory with baseline pods over a fixed hold period and block graduation while canary memory keeps rising. canary
PR-03Run a soak test on the authorization service at production peak request rate for 4 hours and fail when pod memory is still rising in the final hour. load
PR-04Run the new adaptive limiter under sustained Git request load and confirm file server memory returns to baseline after the load stops. load
PR-05Add a test helper that fails a test when goroutines started during the test are still running at its end. unit
PR-06Run the goroutine leak check on every test of the request limiter package. unit

First 30 minutes

FM-01Check pod and node memory for the services changed in the release and look for a steady rise that resets on restart.
FM-02Match the start of the memory growth with the deploy time of each change in the release.
FM-03Drain the canary host if the leak shows there first, before the release reaches the rest of the fleet.
FM-04Roll back the leaking change, and start the manual rollback path at once if rollback tooling depends on the leaking service.
FM-05Restart the affected nodes on a fixed schedule as containment until the revert is deployed.
FM-06Post a status update that lists the degraded features and the error type, such as 404 or 500 responses.
FM-07Confirm memory stays flat after the revert before removing the scheduled restarts.

Download the checklist

Pre-release checklist (Markdown)1.3 KB · v1.0.0
First 30 minutes checklist (Markdown)1.1 KB · v1.0.0

Only in this checklist

Every test class in this checklist also appears in at least one sibling checklist in the group; the item texts differ: Confirm the canary receives at least the per-pod request rate of production peak before an authorization service build graduates.

Questions

What should be checked before a release to prevent Memory leak in new release?

6 pre-release items, PR-01 to PR-06. PR-01: Confirm the canary receives at least the per-pod request rate of production peak before an authorization service build graduates.

Is there a rollback plan for Memory leak in new release?

The first-30-minutes checklist has 7 items. FM-01: Check pod and node memory for the services changed in the release and look for a steady rise that resets on restart.

What should a team do in the first 30 minutes after Memory leak in new release is detected?

The first-30-minutes checklist has 7 items. FM-01: Check pod and node memory for the services changed in the release and look for a steady rise that resets on restart.

Is a Memory leak in new release checklist still needed if deployment is automated?

Yes. Automation runs the steps; the checklist names what the tests must cover: canary and progressive delivery checks, load and performance tests and unit tests.

Which pipeline stages does the Memory leak in new release checklist cover?

The pre-release items map to the post-deploy, pre-deploy and pre-merge stages.

In what formats can the Memory leak in new release checklist be downloaded?

Both checklists download as Markdown files: /downloads/memory-leak-in-new-release-pre-release.md and /downloads/memory-leak-in-new-release-first-30-minutes.md, licensed CC BY 4.0.