feat(trivy): scan running workloads for CVEs and alert on criticals #176

Merged

kblack0610 merged 1 commit from feat/trivy-operator-cve-scanning into master

2026-08-11 17:24:00 +00:00

kblack0610 commented

2026-08-11 17:23:49 +00:00

Owner

Closes the second half of "nothing tells us our pods are stale". Renovate (#172) watches upstream for new versions; nothing looked at what is actually running. There was no Trivy/OSV/Grype/Kyverno anywhere - 58 CRDs in the cluster, not one about vulnerabilities.

Placement (the decision worth reading)

Operator and every scan job pinned to asus-laptop:

node	why not
6x arm64 Pis	control plane + workers, 4-8GB; Trivy DB + per-scan cache would evict real workloads
hp-victus	other amd64 box, but ~70% disk, documented DiskPressure history, no `MIN_FREE_GB` guard
asus-laptop	~12% disk, 64GB RAM

ClientServer, not Standalone

Via operator.builtInTrivyServer (which also sets trivy.serverURL - setting trivy.mode by hand fails to render with .Values.trivy.serverURL is required). The DB is downloaded once by a server StatefulSet instead of by every scan job; Standalone across ~60 images is dozens of multi-hundred-MB pulls per sweep and a fast route to a registry rate limit.

Scanner scope is deliberately narrow

On: image vulnerabilities, exposed secrets, config audit.
Off: infra assessment, RBAC assessment, cluster compliance - those spawn node-collector jobs on every node, Pis included, to answer a CIS-benchmark question nobody asked.

Alerting reuses the existing path

PrometheusRule -> Alertmanager -> ntfy homelab-alerts. CRITICAL only, not any CVE: a homelab fleet always carries a long tail of unfixable lows, and an alert that always fires is an alert nobody reads. A second rule fires if trivy stops reporting at all, because an unscanned cluster and a clean one look identical.

The rule carries release=kube-prometheus-stack - without it the CR applies and is silently never evaluated, the trap that made homelab-app-health inert for its whole existence (#174).

Verification

helm template against chart 0.35.0: 24 objects, scanJob.nodeSelector and all six scanner flags land as intended. kubectl kustomize clean for apps/trivy-operator/, apps/monitoring/, apps/.

Closes the second half of *"nothing tells us our pods are stale"*. Renovate (#172) watches upstream for new versions; nothing looked at **what is actually running**. There was no Trivy/OSV/Grype/Kyverno anywhere - 58 CRDs in the cluster, not one about vulnerabilities. ## Placement (the decision worth reading) Operator **and every scan job** pinned to `asus-laptop`: | node | why not | |---|---| | 6x arm64 Pis | control plane + workers, 4-8GB; Trivy DB + per-scan cache would evict real workloads | | hp-victus | other amd64 box, but **~70% disk**, documented DiskPressure history, no `MIN_FREE_GB` guard | | **asus-laptop** | ~12% disk, 64GB RAM | ## ClientServer, not Standalone Via `operator.builtInTrivyServer` (which also sets `trivy.serverURL` - setting `trivy.mode` by hand fails to render with `.Values.trivy.serverURL is required`). The DB is downloaded once by a server StatefulSet instead of by every scan job; Standalone across ~60 images is dozens of multi-hundred-MB pulls per sweep and a fast route to a registry rate limit. ## Scanner scope is deliberately narrow On: image vulnerabilities, exposed secrets, config audit. Off: infra assessment, RBAC assessment, cluster compliance - those spawn **node-collector jobs on every node**, Pis included, to answer a CIS-benchmark question nobody asked. ## Alerting reuses the existing path PrometheusRule -> Alertmanager -> ntfy `homelab-alerts`. **CRITICAL only**, not any CVE: a homelab fleet always carries a long tail of unfixable lows, and an alert that always fires is an alert nobody reads. A second rule fires if trivy stops reporting at all, because an unscanned cluster and a clean one look identical. The rule carries `release=kube-prometheus-stack` - without it the CR applies and is silently never evaluated, the trap that made `homelab-app-health` inert for its whole existence (#174). ## Verification `helm template` against chart 0.35.0: 24 objects, `scanJob.nodeSelector` and all six scanner flags land as intended. `kubectl kustomize` clean for `apps/trivy-operator/`, `apps/monitoring/`, `apps/`.

kblack0610 added 1 commit

2026-08-11 17:23:49 +00:00

feat(trivy): scan running workloads for CVEs and alert on criticals d94e35c3a6

Closes the second half of "nothing tells us our pods are stale": Renovate
watches upstream for new versions, but nothing looked at what is actually
running. There was no Trivy, OSV, Grype or Kyverno anywhere - 58 CRDs in
the cluster and not one of them about vulnerabilities.

Placement is the decision worth reading. The operator AND every scan job
are pinned to asus-laptop:
  - the six arm64 Pis are the control plane and workers with 4-8GB, and
    Trivy's DB plus per-scan cache would evict real workloads;
  - hp-victus is the other amd64 box but sits at ~70% disk with a
    documented DiskPressure history and no MIN_FREE_GB guard;
  - asus-laptop has ~12% disk and 64GB RAM.

ClientServer mode (via operator.builtInTrivyServer, which also sets
trivy.serverURL - setting trivy.mode by hand fails to render) so the
vulnerability DB is downloaded once by a server StatefulSet rather than by
every scan job. Standalone across ~60 images is dozens of multi-hundred-MB
pulls per sweep and a fast route to a registry rate limit.

Scanner scope is deliberately narrow: image vulnerabilities, exposed
secrets and config audit are on; infra assessment, RBAC assessment and
cluster compliance are off. Those spawn node-collector jobs on EVERY node -
the Pis included - to answer a CIS-benchmark question nobody asked.

Alerting reuses the existing path rather than adding a notifier: a
PrometheusRule -> Alertmanager -> ntfy `homelab-alerts`, the topic actually
subscribed. It alerts on CRITICAL only, not any CVE: a homelab fleet always
carries a long tail of unfixable lows, and an alert that always fires is an
alert nobody reads. A second rule fires if trivy stops reporting at all,
because an unscanned cluster and a clean one look identical.

The rule carries release=kube-prometheus-stack. Without it the CR applies
and is silently never evaluated - the trap that made homelab-app-health
inert for its whole existence (#174).

Verified with `helm template` against chart 0.35.0: renders 24 objects,
scanJob.nodeSelector and all six scanner flags land as intended.