fix(monitoring): one cc:watch_state rule for queue, lint fatal (hc-150) #631

Merged
kblack0610 merged 1 commit from promtool-lint-fatal into master 2026-11-06 22:22:19 +00:00
Owner

Summary

  • apps/monitoring/prometheus-rules-command-center.yaml: the two cc:watch_state{domain=queue} recording rules are now one rule. Weights are unchanged: queue_starved and shepherd_stuck trip at 2, heal_flapping and board_chain at 1, and any of the four going stale gives 1.
  • scripts/test-prometheus-rules.sh: promtool check rules --lint-fatal, so a duplicate rule fails the gate.

assumed: merging into one rule (not deleting one) because the two rules covered different watches, so deleting either would drop a watch from the queue domain.

Verified

Run with promtool v3.8.0 in docker, rules extracted from .spec the same way the script does (yq was not installed locally).

  • master rules + --lint-fatal: command-center FAILED, "lint error 1 duplicate rule(s) found", exit 3.
  • this branch + --lint-fatal: all 6 files SUCCESS (command-center 84 rules, was 85), exit 0.
  • promtool test rules: exit 0, same as master.

Item: promtool-check-rules-prints-lint-error-1
Task: hc-150

## Summary - `apps/monitoring/prometheus-rules-command-center.yaml`: the two `cc:watch_state{domain=queue}` recording rules are now one rule. Weights are unchanged: `queue_starved` and `shepherd_stuck` trip at 2, `heal_flapping` and `board_chain` at 1, and any of the four going stale gives 1. - `scripts/test-prometheus-rules.sh`: `promtool check rules --lint-fatal`, so a duplicate rule fails the gate. assumed: merging into one rule (not deleting one) because the two rules covered different watches, so deleting either would drop a watch from the queue domain. ## Verified Run with promtool v3.8.0 in docker, rules extracted from `.spec` the same way the script does (yq was not installed locally). - master rules + `--lint-fatal`: command-center FAILED, "lint error 1 duplicate rule(s) found", exit 3. - this branch + `--lint-fatal`: all 6 files SUCCESS (command-center 84 rules, was 85), exit 0. - `promtool test rules`: exit 0, same as master. Item: promtool-check-rules-prints-lint-error-1 Task: hc-150
fix(monitoring): one cc:watch_state rule for the queue domain, lint fatal (hc-150)
All checks were successful
checks / ci-disk-prune (pull_request) Successful in 5s
checks / zomboid-scripts (pull_request) Successful in 6s
checks / fleet-roster (pull_request) Successful in 11s
checks / kustomize-builds (pull_request) Successful in 5s
checks / prometheus-rules (pull_request) Successful in 33s
cb62ae4f7d
Two recording rules wrote cc:watch_state{domain=queue}; promtool reported
"lint error 1 duplicate rule(s)" and the two could disagree. Fold them into
one rule that keeps each watch's weight (queue_starved/shepherd_stuck 2,
heal_flapping/board_chain 1), and pass --lint-fatal so a duplicate fails CI.
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kblack0610/home-config!631
No description provided.