Connection pool exhausted: pre-release and first 30 minutes checklists

A new code path leaks or holds connections and the pool runs out under load.

The pre-release checklist maps each item to a test class from the incident page. The first-30-minutes checklist covers detection, containment, the rollback decision and customer communication.

9 pre-release items · 8 first-30-minutes items · verified 2026-10-21

Pre-release checklist

PR-01Run each new background job in the integration suite and confirm the open database connection count returns to its starting value when the job ends. integration
PR-02Exercise every branch of a code path that opens a database connection, including the branch where its condition ends, and assert the connection is closed afterwards. integration
PR-03Count connections checked out per background job with the health-check feature flag on and off, and fail the test when the flag adds a connection. integration
PR-04Run indexing processes through call paths that cross into the older session management code and assert every connection is released. integration
PR-05Run each new query from the release against a production-sized copy of the primary database and fail the run when its execution time exceeds the agreed budget. load
PR-06Run a load test with the new background job on its production schedule alongside user write traffic and confirm primary database pool usage stays below the pool limit. load
PR-07Hold the load test at peak morning traffic until connection counts level off, and fail it when open connections keep rising. load
PR-08Record connection pooler saturation at every feature flag rollout step, including the final step to 100 percent of pods, and halt the rollout when saturation rises above the pre-rollout level. canary
PR-09Halt the rollout when database connections in use per canary pod exceed the baseline pods' count. canary

First 30 minutes

FM-01Read the connection pool and connection pooler saturation for the primary database and count connections that stay open.
FM-02Match the start of the saturation to the latest deploy, a newly enabled feature flag or a newly scheduled background job.
FM-03Disable the feature flag that controls the new code path or background job, even when the link to the flag is not yet proven.
FM-04Halt the new background job directly if no feature flag controls it.
FM-05If no flag or job explains the saturation, revert the deploy that introduced the code holding connections.
FM-06Free stale database connections by hand while the revert rolls out, and confirm the open connection count falls.
FM-07Check for connections that open and never close before spending time on network diagnosis.
FM-08Post a status update that names the failing operations, such as logins, pushes or write requests, and the time the impact began.

Download the checklist

Pre-release checklist (Markdown)2.0 KB · v1.0.0
First 30 minutes checklist (Markdown)1.2 KB · v1.0.0

Only in this checklist

Every test class in this checklist also appears in at least one sibling checklist in the group; the item texts differ: Run each new background job in the integration suite and confirm the open database connection count returns to its starting value when the job ends.

Questions

What should be checked before a release to prevent Connection pool exhausted?

9 pre-release items, PR-01 to PR-09. PR-01: Run each new background job in the integration suite and confirm the open database connection count returns to its starting value when the job ends.

Is there a rollback plan for Connection pool exhausted?

The first-30-minutes checklist has 8 items. FM-01: Read the connection pool and connection pooler saturation for the primary database and count connections that stay open.

What should a team do in the first 30 minutes after Connection pool exhausted is detected?

The first-30-minutes checklist has 8 items. FM-01: Read the connection pool and connection pooler saturation for the primary database and count connections that stay open.

Is a Connection pool exhausted checklist still needed if deployment is automated?

Yes. Automation runs the steps; the checklist names what the tests must cover: integration tests, load and performance tests and canary and progressive delivery checks.

Which pipeline stages does the Connection pool exhausted checklist cover?

The pre-release items map to the pre-deploy and post-deploy stages.

In what formats can the Connection pool exhausted checklist be downloaded?

Both checklists download as Markdown files: /downloads/connection-pool-exhausted-pre-release.md and /downloads/connection-pool-exhausted-first-30-minutes.md, licensed CC BY 4.0.