fix(ansible-runner): one down host must not stop the whole convergence check #197

Merged

kblack0610 merged 1 commit from fix/convergence-check-resilient into master

2026-08-17 19:08:40 +00:00

kblack0610 commented

2026-08-17 19:08:34 +00:00

Owner

A down machine was breaking drift detection for the entire fleet.

The check was --limit thinkcentre - that was its entire scope. Two consequences:

hp-victus and asus-laptop were never drift-checked by anything, including the k3s-node-no-suspend role whose entire purpose is preventing the node loss that happened on 2026-08-17 (and previously 2026-05-07, documented in that role s README).
thinkcentre has been off-network since at least 2026-08-06, so the job failed nightly while checking nothing at all.

Now it probes each host, checks everything reachable, and names what it skipped - an unchecked host must not look like a clean one. It errors only when nothing is reachable.

Scope widened to thinkcentre, hp-victus, asus-laptop.

Verified against a stubbed ssh-keyscan:

Scenario	Result
all up	checks all three
thinkcentre down (the break)	warns, still checks both laptops
thinkcentre + hp-victus down (today)	warns, still checks asus-laptop
all down	errors rather than silently passing

Does not fix the outage itself - hp-victus still needs a physical power-on - but it means the guard that should have prevented it is actually verified from now on.

**A down machine was breaking drift detection for the entire fleet.** The check was `--limit thinkcentre` - that was its *entire* scope. Two consequences: 1. `hp-victus` and `asus-laptop` were **never drift-checked by anything**, including the `k3s-node-no-suspend` role whose entire purpose is preventing the node loss that happened on 2026-08-17 (and previously 2026-05-07, documented in that role s README). 2. thinkcentre has been off-network since at least 2026-08-06, so the job failed nightly while checking nothing at all. Now it probes each host, checks everything reachable, and **names what it skipped** - an unchecked host must not look like a clean one. It errors only when nothing is reachable. Scope widened to `thinkcentre`, `hp-victus`, `asus-laptop`. **Verified** against a stubbed `ssh-keyscan`: | Scenario | Result | |---|---| | all up | checks all three | | thinkcentre down (the break) | warns, still checks both laptops | | thinkcentre + hp-victus down (today) | warns, still checks asus-laptop | | all down | errors rather than silently passing | Does not fix the outage itself - hp-victus still needs a physical power-on - but it means the guard that should have prevented it is actually verified from now on.

kblack0610 added 1 commit

2026-08-17 19:08:34 +00:00

fix(ansible-runner): one down host must not stop the whole convergence check c212c69620

The check was --limit thinkcentre. That was its entire scope, so the laptops were
never drift-checked by anything - including the k3s-node-no-suspend guard on
hp-victus, whose whole purpose is preventing the exact node loss that happened on
2026-08-17 (and before it on 2026-05-07). thinkcentre has also been off-network
since at least 2026-08-06, so the job failed nightly while checking nothing.

Now it probes each host, checks everything reachable, and names what it skipped.
It only errors when nothing at all can be reached. A partially available fleet is
normal; a drift check that gives up on the first missing machine is not a check.

Scope widened to thinkcentre, hp-victus and asus-laptop.

Verified against a stubbed ssh-keyscan: all-up checks all three; thinkcentre down
warns and still checks both laptops; two down still checks the third; all down
errors rather than silently passing.