# Background job queue backlog: pre-release checklist

Source: https://releasefailures.com/incidents/background-job-queue-backlog/
Content version: 1.0.0
Verified on: 2026-10-21
Licence: CC BY 4.0

Definition: A job change slows processing and the queue grows faster than it drains.
- [ ] PR-01 After a major-version upgrade of any shared library, start a background worker in an integration environment and confirm each scheduler thread is still running after startup. (test class: Integration tests)
- [ ] PR-02 Add an integration test that registers a recurring job and asserts it is enqueued within one schedule interval after a worker starts. (test class: Integration tests)
- [ ] PR-03 List every consumer of an upgraded library, including third-party job libraries, and run each consumer's tests against the new version. (test class: Integration tests)
- [ ] PR-04 Run each new periodic task's query against a production-sized copy of its table and confirm it finishes before the next scheduled run. (test class: Load and performance tests)
- [ ] PR-05 Run the infrastructure management workers under the expected scheduling workload after a dependency upgrade and compare task throughput with the previous release. (test class: Load and performance tests)
- [ ] PR-06 Confirm in a load test that worker scaling adds workers when queue depth rises. (test class: Load and performance tests)
- [ ] PR-07 Compare queue depth and the age of the oldest pending task with baseline before promoting a release past the canary step. (test class: Canary and progressive delivery checks)
- [ ] PR-08 Hold the canary until every recurring job has run once within its expected interval on the new release. (test class: Canary and progressive delivery checks)