Background job queue backlog: symptoms, cause, and the test that catches it

Symptoms

Root causes

1. stage: code A routine framework update moved a connection pool library from 2.5.5 to 3.0.2 with a keyword-only API, and three scheduler threads per worker of the job library failed on startup.[1]
2. stage: code A dependency upgrade in the infrastructure management application caused a timing lock issue that greatly reduced task worker throughput.[2]
3. stage: infra The task workers failed to scale up for the increased workload scheduling, so the work queue built up.[2]
4. stage: code A periodic task deployed on 2022-06-23 ran a slow row-counting query every 5 minutes, and its long-running instances used up the workers available for other tasks.[3]

Public examples

1. [1]Buildkite, 2026-08-10. A routine Rails update pulled a new major version of connection_pool that Sidekiq could not use; internal Sidekiq scheduler threads stopped on startup, so recurring background work stopped and queued jobs waited longer for dispatch until the library was pinned back. primary Latency on Pipelines
2. [2]Heroku, 2022-09-23. A change to the application that manages Common Runtime infrastructure included a dependency upgrade that greatly reduced task worker throughput; the work queue built up, new router instances were not initialised correctly and were deleted, and requests to dynos failed until the change was rolled back. primary Degraded availability
3. [3]PyPI, 2022-06-28. A deployed periodic task for 2FA statistics ran a slow row-counting query every 5 minutes; over several days these tasks occupied more of the workers until the central task queue held nearly 50K pending tasks and CDN purges and emails were delayed by more than 14 hours. primary Queue delay affecting CDN purges and emails

Test classes that catch it

Risk

Pre-release checklist

• PR-01After a major-version upgrade of any shared library, start a background worker in an integration environment and confirm each scheduler thread is still running after startup. integration
• PR-02Add an integration test that registers a recurring job and asserts it is enqueued within one schedule interval after a worker starts. integration
• PR-03List every consumer of an upgraded library, including third-party job libraries, and run each consumer's tests against the new version. integration
• PR-04Run each new periodic task's query against a production-sized copy of its table and confirm it finishes before the next scheduled run. load
• PR-05Run the infrastructure management workers under the expected scheduling workload after a dependency upgrade and compare task throughput with the previous release. load
• PR-06Confirm in a load test that worker scaling adds workers when queue depth rises. load
• PR-07Compare queue depth and the age of the oldest pending task with baseline before promoting a release past the canary step. canary
• PR-08Hold the canary until every recurring job has run once within its expected interval on the new release. canary

What catches this: catalogue entries

Listed because their own documentation or site names integration tests, load and performance tests, canary and progressive delivery checks on the verification date. Alphabetical. Not a recommendation.

Related incidents

• Connection pool exhausted A new code path leaks or holds connections and the pool runs out under load.
• Memory leak in new release Memory grows per request or per screen until processes restart or the app is killed.
• Thundering herd on restart Clients reconnect simultaneously after a rolling restart and overload a dependency.
• Cache invalidation storm A deploy flushes or invalidates caches at once and origin load spikes.
• Third-party rate limit hit A new call pattern exceeds a provider's quota and requests are rejected.

Questions