fix(flux-audit): stop a working alarm from looking like a broken job #173

Merged

kblack0610 merged 1 commit from fix/flux-audit-alarm-semantics into master

2026-08-11 17:10:50 +00:00

kblack0610 commented

2026-08-11 17:10:38 +00:00

Owner

flux-audit was never broken. It has been running every night, finding real orphans, and exiting 1 to report them - which is indistinguishable from a crashed job. So three months of BackoffLimitExceeded read as noise.

Confirmed by running it: 7 orphans right now, 5 of them cert-manager resources Flux does not own.

cert-manager   deployment/cert-manager
cert-manager   deployment/cert-manager-cainjector
cert-manager   deployment/cert-manager-webhook
cert-manager   service/cert-manager
cert-manager   service/cert-manager-webhook
langfuse       persistentvolumeclaim/valkey-data-langfuse-redis-primary-0
monitoring     configmap/nodejs-dashboard

Worse: it posted findings to ntfy topic flux-audit, which nothing subscribes to. Alertmanager sends everything to homelab-alerts. Every finding went into the void twice - an alarm nobody received, expressed as a failure nobody trusted.

Changes

Exit-code contract. exit 0 = the audit RAN (findings go out by ntfy); exit 1 = the audit is BROKEN. Job failure now means the check is blind, which is the only thing worth paging on.
Notify homelab-alerts, and treat an undeliverable notification as a job failure. A silent alarm is the bug being fixed, so it must not pass quietly.
restartPolicy: Never. OnFailure makes the Job controller delete the failed pod - which is exactly why months of failures could only be seen as "no pods retained, no logs", and why diagnosing this needed a hand-built debug pod.

New: prometheus-rules-jobs.yaml

So no scheduled check can go quiet again:

HomelabCronJobFailing - would fire right now for 11 jobs, including immich-originals-backup and ansible-runner/convergence-check, both failing since ~2026-08-06 with nobody told.
HomelabCronJobNotScheduling - a job that stops being scheduled leaves no failed job to alert on. Measured against each CronJob's own next_schedule_time, not a flat "not run in 48h": nas/ runs four weekly jobs, and a flat threshold would have paged on all four every week for being healthy - which is how an alert teaches people to ignore it.

Verification

Both PromQL expressions run against live Prometheus: 0 false positives on the weekly jobs, suspended CronJobs correctly excluded.
All three script paths tested in-cluster: orphans+ntfy-ok -> exit 0 (real ntfy message id returned), orphans+ntfy-down -> exit 1, audit-crash -> exit 1.
kubectl kustomize clean for apps/flux-audit/, apps/monitoring/, apps/.

The 7 orphans themselves are left alone - that is a separate cleanup decision, not a monitoring fix.

**flux-audit was never broken.** It has been running every night, finding real orphans, and exiting 1 to report them - which is indistinguishable from a crashed job. So three months of `BackoffLimitExceeded` read as noise. Confirmed by running it: **7 orphans right now**, 5 of them cert-manager resources Flux does not own. ``` cert-manager deployment/cert-manager cert-manager deployment/cert-manager-cainjector cert-manager deployment/cert-manager-webhook cert-manager service/cert-manager cert-manager service/cert-manager-webhook langfuse persistentvolumeclaim/valkey-data-langfuse-redis-primary-0 monitoring configmap/nodejs-dashboard ``` Worse: it posted findings to ntfy topic `flux-audit`, which nothing subscribes to. Alertmanager sends everything to `homelab-alerts`. Every finding went into the void twice - an alarm nobody received, expressed as a failure nobody trusted. ## Changes - **Exit-code contract.** `exit 0` = the audit RAN (findings go out by ntfy); `exit 1` = the audit is BROKEN. Job failure now means the check is blind, which is the only thing worth paging on. - **Notify `homelab-alerts`**, and treat an undeliverable notification as a job failure. A silent alarm is the bug being fixed, so it must not pass quietly. - **`restartPolicy: Never`.** `OnFailure` makes the Job controller *delete* the failed pod - which is exactly why months of failures could only be seen as "no pods retained, no logs", and why diagnosing this needed a hand-built debug pod. ## New: prometheus-rules-jobs.yaml So no scheduled check can go quiet again: - `HomelabCronJobFailing` - **would fire right now for 11 jobs**, including `immich-originals-backup` and `ansible-runner/convergence-check`, both failing since ~2026-08-06 with nobody told. - `HomelabCronJobNotScheduling` - a job that stops being scheduled leaves no failed job to alert on. Measured against each CronJob's **own `next_schedule_time`**, not a flat "not run in 48h": `nas/` runs four *weekly* jobs, and a flat threshold would have paged on all four every week for being healthy - which is how an alert teaches people to ignore it. ## Verification - Both PromQL expressions run against live Prometheus: 0 false positives on the weekly jobs, suspended CronJobs correctly excluded. - All three script paths tested in-cluster: orphans+ntfy-ok -> exit 0 (real ntfy message id returned), orphans+ntfy-down -> exit 1, audit-crash -> exit 1. - `kubectl kustomize` clean for `apps/flux-audit/`, `apps/monitoring/`, `apps/`. > The 7 orphans themselves are left alone - that is a separate cleanup decision, not a monitoring fix.

kblack0610 added 1 commit

2026-08-11 17:10:38 +00:00

fix(flux-audit): stop a working alarm from looking like a broken job 53cddbf284

flux-audit was not broken. It has been running every night, finding real
orphans, and exiting 1 to report them - which is indistinguishable from a
crashed job, so three months of "BackoffLimitExceeded" read as noise and
nobody acted on it. Confirmed by running it: 7 orphans right now, 5 of them
cert-manager resources Flux does not own.

Worse, it posted its findings to ntfy topic `flux-audit`, which nothing
subscribes to. Alertmanager sends everything to `homelab-alerts`
(apps/monitoring/helm-values.yaml). So every finding went into the void
twice over: an alarm nobody received, expressed as a failure nobody trusted.

Three changes:

- Exit-code contract. exit 0 = the audit RAN (findings go out by ntfy);
  exit 1 = the audit is BROKEN. A failure now means the check is blind,
  which is the only thing worth paging a human about.
- Notify `homelab-alerts`, the topic a phone is actually subscribed to, and
  treat an undeliverable notification as a job failure - a silent alarm is
  the failure mode being fixed here, so it must not pass quietly.
- restartPolicy: Never, not OnFailure. OnFailure makes the Job controller
  DELETE the failed pod, which is why months of failures could only be seen
  as "no pods retained" with no logs. This is what made diagnosis need a
  hand-built debug pod.

Adds prometheus-rules-jobs.yaml so no scheduled check can go quiet again:

- HomelabCronJobFailing - fires on any failed job. Would fire right now for
  11, including immich-originals-backup and ansible-runner's
  convergence-check, both failing since ~2026-08-06 with nobody told.
- HomelabCronJobNotScheduling - a job that stops being scheduled leaves no
  failed job to alert on. Measured against each CronJob's own
  next_schedule_time rather than a flat "not run in 48h": nas/ runs four
  WEEKLY jobs, and a flat threshold would have paged on all four every week
  for being perfectly healthy, which is how an alert teaches people to
  ignore it.

Both expressions validated against live Prometheus (0 false positives on
the weekly jobs, suspended CronJobs correctly excluded), and all three
script paths tested in-cluster: orphans+ntfy-ok -> 0, orphans+ntfy-down
-> 1, audit-crash -> 1.