Memory leak in new release: symptoms, cause, and the test that catches it

Symptoms

Root causes

1. stage: code A performance and resilience optimisation to an authorization microservice contained a memory leak that high traffic exposed.[1]
2. stage: release-process The canary ring did not send enough traffic to the authorization service to reveal the leak, so the build graduated to production.[1]
3. stage: code A new adaptive limiter in the Git storage service created a memory leak on file server nodes.[2]
4. stage: code A change to the Git storage service introduced a goroutine leak, and one merge request was identified as the most likely cause.[3]

Public examples

1. [1]GitHub, 2023-11-03. A performance and resilience optimisation to GitHub's authorization microservice contained a memory leak exposed under high traffic; pods crashed repeatedly, authorization checks failed and pages returned 404s until the change was rolled back. primary Incident with Git Operations, Issues, Pull Requests, Actions, API Requests, Codespaces, Packages, Pages and Webhooks
2. [2]GitLab, 2023-08-16. A new adaptive limiter deployed in Gitaly caused memory to build up on a GitLab file server from 14 August 2023 until specific Gitaly nodes became unresponsive and 3.35 percent of Git requests returned 500 errors; the commit was reverted. primary 2023-08-16: Connection errors on gitaly causing 500s (#16187)
3. [3]GitLab, 2021-09-21. After a deploy to the GitLab canary file server, a Gitaly goroutine leak caused memory to grow; a Gitaly merge request was identified as the likely cause and the server was rolled back to the previous package version. primary 2021-09-21 Gitaly goroutine memory leak on file-cny-01 since last deploy (#5566)

Test classes that catch it

Risk

Pre-release checklist

• PR-01Confirm the canary receives at least the per-pod request rate of production peak before an authorization service build graduates. canary
• PR-02Compare canary pod memory with baseline pods over a fixed hold period and block graduation while canary memory keeps rising. canary
• PR-03Run a soak test on the authorization service at production peak request rate for 4 hours and fail when pod memory is still rising in the final hour. load
• PR-04Run the new adaptive limiter under sustained Git request load and confirm file server memory returns to baseline after the load stops. load
• PR-05Add a test helper that fails a test when goroutines started during the test are still running at its end. unit
• PR-06Run the goroutine leak check on every test of the request limiter package. unit

What catches this: catalogue entries

Listed because their own documentation or site names canary and progressive delivery checks, load and performance tests, unit tests on the verification date. Alphabetical. Not a recommendation.

Related incidents

• Connection pool exhausted A new code path leaks or holds connections and the pool runs out under load.
• Background job queue backlog A job change slows processing and the queue grows faster than it drains.
• Cache invalidation storm A deploy flushes or invalidates caches at once and origin load spikes.
• Thundering herd on restart Clients reconnect simultaneously after a rolling restart and overload a dependency.
• Cold start latency regression Serverless or container cold starts become slower after a dependency or image change.

Questions