Release Incident Playbook

Disk full from new log volume: symptoms, cause, and the test that catches it

A logging change fills disks and services stop writing.

Disk full from new log volume is documented by 3 accepted public reports from 3 organisations, 2 of them published in the last 5 years.

  • Likelihood: low (2 public reports in 5 years)
  • Impact: medium (2 of 3 reports: availability)

3 public reports · verified 2026-10-22

Symptoms

User-visible

  • Pages within the hosted sites failed to load intermittently after an update meant to improve system monitoring.[2]
  • Users saw error messages and 500 error pages on some page loads for about 38 minutes.[2]
  • Sporadic partial outages with elevated response latency ran from 9:15 to 9:44 am CDT, the morning after an operating system update.[3]
  • Some requests failed with client side and server side timeouts.[3]
  • One commenter reports that 100GB of repeated errors from a new discovery service filled the firewall disk and took the whole network out twice in a day.[1]

Monitoring-visible

  • Servers reported high disk space utilization at 10:19 MT, ten minutes before client impact began.[2]
  • Root partitions on the application servers went from 65% to 90% full over about 8 hours, depending on traffic.[3]
  • The disk health check replaced every server above 90% usage, and many servers crossed that threshold at once.[3]
  • High API latency alarms and a downtime alarm fired, and the remaining servers showed high CPU usage.[3]
  • Developers identified the low disk space at 8:33 am CDT, before the latency alarm at 9:15 and the downtime alarm at 9:19.[3]

Root causes

  1. stage: code An application update intended to improve system monitoring generated more log files than anticipated, and the extra logging filled server disk space.[2]
  2. stage: infra An operating system update deployed to the application servers used more base disk space and produced a much greater volume of kernel audit logs.[3]
  3. stage: config The log rotation configuration did not keep pace with the new kernel audit log volume, so root partition usage grew slowly until servers could not service requests.[3]
  4. stage: infra The disk health check removed servers above 90% usage, and many servers crossed that threshold together during morning traffic.[3]

Public examples

  1. [1]OPNsense, 2026-01-16. After an OPNsense update that added the new hostwatch automatic discovery service, hostwatch logged the same libpcap error several times per second when virtual interfaces such as WireGuard were destroyed and recreated, filling firewall disks and upstream syslog hosts until maintainers shipped hostwatch 1.0.5, which disabled two offending log messages. primary ERROR hostwatch: Error reading from capture: libpcap error: The interface disappeared · Issue #8 · opnsense/hostwatch
  2. [2]Benevity, 2024-09-18. An update to Benevity Spark meant to enhance system monitoring generated a larger volume of log files than anticipated, servers reported high space utilization, and users saw intermittent 500 error pages for about 38 minutes. primary Benevity Status - Spark unavailable
  3. [3]SCORM Cloud, 2021-09-24. An operating system update deployed to SCORM Cloud application servers produced far more Linux kernel audit logs, and with insufficient log rotation the root partitions filled until the disk health check removed many servers at once during morning traffic, causing timeouts and partial outages. primary SCORM Cloud Status - Slow Response Times

Test classes that catch it

Test classStageLimitationBasis
Load and performance testspre-deployA load test shorter than the roughly 8 hours the root partitions took to fill from 65% to 90% ends before the disk fills, so the kernel audit log growth passes.by design analysis
Canary and progressive delivery checkspost-deployA canary judged only on error rate passes, because root partition usage after the operating system update climbed for about 8 hours before the first timeout.by design analysis
Chaos and resilience testspre-deployA chaos test of the health check finds the mass replacement of servers at 90% disk usage but does not reveal that a logging or operating system change caused the growth.by design analysis

Risk

  • Likelihood: low (2 public reports in 5 years)
  • Impact: medium (2 of 3 reports: availability)
Counts behind the grades
Accepted public reports3
Published in the last 5 years2
Impact stated: availability2
Impact stated: degraded1

How the grades are computed

Pre-release checklist

  • PR-01
  • PR-02
  • PR-03
  • PR-04
  • PR-05
  • PR-06
  • PR-07

First 30 minutes

  • FM-01
  • FM-02
  • FM-03
  • FM-04
  • FM-05
  • FM-06
  • FM-07
  • FM-08

Download the checklist

What catches this: catalogue entries

Listed because their own documentation or site names load and performance tests, canary and progressive delivery checks, chaos and resilience tests on the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Providers whose site names these test classes

Only on this page

Report [3] records a time to detect of 1366 minutes for disk full from new log volume. The operating system update was deployed on September 14, and developers first identified the low disk space at 8:33 am CDT on September 15 [3]. Root partitions had climbed from 65% to 90% over about 8 hours, and the disk health check then removed many servers at once [3].

Report [3] states that the slow leak passed the usual quality assurance tests because it needed many hours of production traffic to trigger failures. In report [2], high disk utilization appeared only ten minutes before client impact, which leaves a far shorter window than the overnight growth in report [3].

Questions

What causes disk full from new log volume?

In report [2], an update meant to improve monitoring generated more log files than anticipated. In report [3], an operating system update produced far more kernel audit logs than the log rotation configuration could clear. In report [1], a new discovery service logged the same capture error several times per second whenever a virtual interface was recreated.

What are the symptoms of disk full from new log volume?

Users saw intermittent 500 error pages for about 38 minutes in report [2]. In report [3], root partitions went from 65% to 90% full over about 8 hours before latency and downtime alarms fired.

Why did tests not catch disk full from new log volume?

Report [3] states the slow leak passed the usual quality assurance tests because it needed many hours of production traffic. A load test shorter than that window ends before the disk fills.

Which test class catches disk full from new log volume, and what does it miss?

Load and performance tests, canary checks and chaos tests carry the by-design label, because none of the three reports names a test class. A canary judged only on error rate misses the hours of disk growth before the first error.

How long did it take to detect disk full from new log volume in the public reports?

Report [3] records a time to detect of 1366 minutes. Report [2] gives no time to detect, but disk alerts there came ten minutes before client impact.

Should a team roll back or fix forward when disk full from new log volume happens?

In report [2], the affected server instances were replaced and the logging level was reduced. In report [3], an engineer scaled out the server stack manually, followed by larger base disks and more aggressive log rotation. In report [1], maintainers fixed forward with a release that disabled two offending log messages.

Release Incident Playbook (2026). Disk full from new log volume: symptoms, cause, and the test that catches it, content version 1.0.0. https://releasefailures.com/incidents/disk-full-from-new-log-volume/. Accessed 2026-10-21.