fix(monitoring): add the release label PrometheusRules need to be evaluated #174

Merged

kblack0610 merged 1 commit from fix/prometheusrule-release-label into master

2026-08-11 17:13:23 +00:00

kblack0610 commented

2026-08-11 17:13:11 +00:00

Owner

The Prometheus CR selects on ruleSelector.matchLabels.release=kube-prometheus-stack. A PrometheusRule without that label applies cleanly, appears in kubectl get prometheusrule, and is then silently ignored forever. There is no error anywhere to notice.

Two rulesets were in that state:

homelab-scheduled-jobs - added in #173 minutes ago. Caught because the alerts it should have fired for 11 failing jobs never appeared.
homelab-app-health - inert for its entire existence. HomelabAppDown, HomelabAppDegraded, HomelabAppRestartSpike and six recording rules. Nothing has been watching for "a deployment has had zero ready capacity for 10 minutes"; the alert exists in git and does not exist in Prometheus.

Verification

Against the live /api/v1/rules endpoint - loaded groups were homelab-node-health.alerts and litellm-cost-control.alerts and neither of the above. Those two carry the label, which is why they work. All four rule files in the repo now carry it.

Same disease as #173: a check that fails by doing nothing, and looks healthy from every angle except the one nobody checks.

The Prometheus CR selects on `ruleSelector.matchLabels.release=kube-prometheus-stack`. A PrometheusRule without that label **applies cleanly, appears in `kubectl get prometheusrule`, and is then silently ignored forever.** There is no error anywhere to notice. Two rulesets were in that state: - **`homelab-scheduled-jobs`** - added in #173 minutes ago. Caught because the alerts it should have fired for 11 failing jobs never appeared. - **`homelab-app-health`** - inert for its entire existence. `HomelabAppDown`, `HomelabAppDegraded`, `HomelabAppRestartSpike` and six recording rules. Nothing has been watching for *"a deployment has had zero ready capacity for 10 minutes"*; the alert exists in git and does not exist in Prometheus. ## Verification Against the live `/api/v1/rules` endpoint - loaded groups were `homelab-node-health.alerts` and `litellm-cost-control.alerts` and neither of the above. Those two carry the label, which is why they work. All four rule files in the repo now carry it. Same disease as #173: a check that fails by doing nothing, and looks healthy from every angle except the one nobody checks.

kblack0610 added 1 commit

2026-08-11 17:13:11 +00:00

fix(monitoring): add the release label PrometheusRules need to be evaluated 482fc9ba91

The Prometheus CR selects rules with
ruleSelector.matchLabels.release=kube-prometheus-stack. A PrometheusRule
without that label applies cleanly, shows up in `kubectl get
prometheusrule`, and is then silently ignored forever - there is no error
anywhere to notice.

Two rulesets were in that state:

- homelab-scheduled-jobs, added minutes ago in #173. Caught because the
  alerts it should have fired for 11 failing jobs never appeared.
- homelab-app-health, which has been inert the whole time it has existed:
  HomelabAppDown, HomelabAppDegraded, HomelabAppRestartSpike and six
  recording rules. Nothing has been watching for "a deployment has had zero
  ready capacity for 10 minutes" - the alert exists in git and does not
  exist in Prometheus.

Verified against the live rule API: before, Prometheus had loaded
homelab-node-health.alerts and litellm-cost-control.alerts but neither of
the above; those two carry the label, which is why they work.

Same disease as #173 - a check that fails by doing nothing, and looks
healthy from every angle except the one nobody checks.