How to read the risk map before choosing a tool or a contractor

The likelihood grade counts accepted primary and secondary reports from distinct organisations published in the last 5 years.

Six or more reports give high, 3 to 5 reports give medium, and 2 reports give low.

The label prints the count behind the grade, for example "Likelihood: low (2 public reports in 5 years)".

A page with only 1 report inside the 5-year window prints "Likelihood: not published (1 public report in 5 years)".

Nine of the 58 published incidents carry that label.

The model does not weight by company size, industry or the author's opinion.

The model does not produce a single score.

The impact grade uses the impact class each accepted report states: availability, data, money, compliance or degraded.

A majority of reports in data, money or compliance gives high.

A majority in availability gives medium.

A majority in degraded gives low.

When fewer than 2 reports state an impact, the label reads "Impact: unrated (fewer than 2 reports state impact)".

Across the 58 published incidents, 40 grade medium, 11 grade high, 5 grade low and 2 are unrated.

Group pages and the incident index draw a 3x3 grid of likelihood against impact, with unrated impact in a separate row.

Each caught-by row names a test class, the pipeline stage where the class runs, one limitation sentence and a basis label.

The label "cited" means an accepted report for that incident names the test class.

The label "by design analysis" means the link between the incident and the class is the author's reasoning.

The 58 published incidents carry 158 caught-by rows: 17 cited and 141 by design analysis.

Fifteen of the 58 incidents have at least one cited row.

The limitation sentence states what the class misses for that specific incident.

The What catches this block lists tools and providers for the test classes in the caught-by table of the same page.

A tool appears under a class only when a page in the tool's own documentation describes that class.

A provider appears under a class only when a page on the provider's own site names the matching service.

Marketing pages, reviews and blogs do not create a listing.

Every row carries its source URL and a verification date.

Rows are sorted alphabetically, and the order is not a ranking.

The block does not recommend any entry.

When no entry documents a class, the block prints "No tool in the catalogue documents this test class."

In the current capability matrix, 89 tools document at least one test class.

Thirty-two tools document end-to-end tests (user flows through the real UI against a deployed environment).

Eight tools document migration tests (migrations run against a production-shaped copy with timing and rollback checks).

Ten tools document load and performance tests (traffic replay or synthetic load with latency and resource budgets).

Seven tools document canary and progressive delivery checks (a small share of traffic runs the new version with automated metric comparison).

Five tools document configuration and environment tests (checks that config, flags, secrets and endpoints resolve correctly per environment).

In the current capability matrix, 60 providers name at least one test class.

No provider names canary and progressive delivery checks, configuration and environment tests or synthetic monitoring (scripted checks of key user flows against production on a schedule).

For those three classes, the provider list prints "No provider in the catalogue names this test class on its site."

Memory leak in new release grades "Likelihood: low (2 public reports in 5 years)" and "Impact: medium (2 of 2 reports: availability)".

The memory leak page holds 3 accepted reports, and 1 of them falls outside the 5-year window.

In a status page report from 2023, the canary ring did not send enough traffic to an authorization service to expose a memory leak [1].

That build graduated to production at 6:37 PM UTC, and alerts triggered at 6:44 PM UTC [1].

The canary row on the memory leak page is labelled cited because report [1] names the canary ring.

The capability matrix lists 7 tools and no provider for canary and progressive delivery checks.

The load and unit rows on the memory leak page carry the by design analysis label.

The matrix lists 10 tools for load and performance tests and 9 tools for unit tests (tests of one function or class in isolation).

Irreversible migration fails midway grades "Likelihood: low (2 public reports in 5 years)" and "Impact: medium (2 of 4 reports: availability)".

Of its 4 accepted reports, 2 state availability, 1 states degraded and 1 states data.

In an engineering post-mortem (a written review of an incident: timeline, cause, impact and follow-up actions) from 2023, a migration that added a unique UUID field failed partway after new records were written during a backfill of about 400k rows [2].

That report states that no load testing ran against the migration [2].

That report also states that load testing would have had to run while UUID generation was running [2].

In a post-mortem from 2017, a release script completed the schema changes but failed to move data into the new tables [3].

That report names enhanced testing of data migration during schema updates as a follow-up [3].

The migration row and the load row on the irreversible migration page are both labelled cited.

The migration limitation states that migration tests check the schema change itself, not the application code paths that read the new schema.

The matrix lists 8 tools for migration tests and 10 tools for load and performance tests.

A tool listed under migration tests does not close the concurrent-write gap in report [2] unless load runs during the migration.

The steps below read one incident page from the risk card down to the catalogue rows.