feat(monitoring): alert on node DiskPressure + root-fs filling up #109

Merged

kblack0610 merged 1 commit from fix/node-diskpressure-alert into master

2026-07-09 19:21:01 +00:00

kblack0610 commented

2026-07-09 19:20:58 +00:00

Owner

Follow-up to #108. Adds NodeDiskPressure (critical) + NodeRootFilesystemFillingUp (warning, <15% free) so the 2026-07-09 failure mode pages next time. Validated both PromQL exprs against live Prometheus.

kblack0610 added 1 commit

2026-07-09 19:20:58 +00:00

feat(monitoring): alert on node DiskPressure + root-fs filling up efd8003dcd

The 2026-07-09 outage (forgejo backups filled asus-laptop's root disk ->
DiskPressure -> whole workload cluster evicted) paged no one: the standard
node-exporter filesystem alerts route to ntfy, but alertmanager runs on
asus-laptop itself and was evicted by the very pressure it should report.

Adds two homelab-node-health rules:
- NodeDiskPressure (critical, 2m): kube_node_status_condition DiskPressure=true.
  The fast, direct signal for this failure mode.
- NodeRootFilesystemFillingUp (warning, 30m): root fs < 15% free. The disk
  grew ~55G/day for weeks; this would have paged ~5 days ahead.

In-cluster alerting can be evicted by the pressure it reports, so an
out-of-cluster Sentinel watch is the belt-and-suspenders (separate change).