fix(monitoring): drop HomelabCronJobFailing, a duplicate of KubeJobFailed #175

Merged

kblack0610 merged 1 commit from fix/drop-duplicate-job-alert into master

2026-08-11 17:17:11 +00:00

kblack0610 commented

2026-08-11 17:16:55 +00:00

Owner

Self-correction on #173. The bundled kube-prometheus-stack rule KubeJobFailed already alerts on kube_job_failed > 0 with the same 15m window and routes to the same ntfy topic - shipping mine would have paged twice for every failed job.

Worth recording why the duplicate looked necessary: KubeJobFailed has been firing this whole time, continuously, for flux-audit - because that job reported findings by exiting 1. A nightly alert for something that is working is how alert fatigue starts, and it is the actual reason nobody acted on flux-audit for months. The fix belongs at the source (#173, the exit-code contract), not in a new rule.

HomelabCronJobNotScheduling stays - nothing bundled covers a CronJob that silently stops being scheduled, which leaves no failed job to alert on at all.

Verified against the live rules API: KubeJobFailed expr kube_job_failed{job="kube-state-metrics"} > 0, for: 900 - functionally identical to what I had added.

Self-correction on #173. The bundled kube-prometheus-stack rule `KubeJobFailed` already alerts on `kube_job_failed > 0` with the same 15m window and routes to the same ntfy topic - shipping mine would have **paged twice for every failed job**. Worth recording why the duplicate looked necessary: `KubeJobFailed` **has been firing this whole time**, continuously, for `flux-audit` - because that job reported findings by exiting 1. A nightly alert for something that is working is how alert fatigue starts, and it is the actual reason nobody acted on flux-audit for months. The fix belongs at the source (#173, the exit-code contract), not in a new rule. `HomelabCronJobNotScheduling` stays - nothing bundled covers a CronJob that silently stops being scheduled, which leaves no failed job to alert on at all. Verified against the live rules API: `KubeJobFailed` expr `kube_job_failed{job="kube-state-metrics"} > 0`, `for: 900` - functionally identical to what I had added.

kblack0610 added 1 commit

2026-08-11 17:16:55 +00:00

fix(monitoring): drop HomelabCronJobFailing, a duplicate of KubeJobFailed d3dfc03073

The bundled kube-prometheus-stack rule KubeJobFailed already alerts on
kube_job_failed > 0 with the same 15m window, and routes to the same ntfy
topic. Shipping a second copy in #173 would have paged twice for every
failed job.

Worth recording why the duplicate looked necessary: KubeJobFailed HAS been
firing this whole time, continuously, for flux-audit - because that job
reported its findings by exiting 1. A nightly alert for something that is
working is how alert fatigue starts, and it is the actual reason nobody
acted on flux-audit for months. The fix belongs at the source (#173, the
exit-code contract), not in a new rule.

HomelabCronJobNotScheduling stays: nothing bundled covers a CronJob that
silently stops being scheduled, which leaves no failed job to alert on.