kblack0610/home-config
1
0
Fork 0
Code Issues Pull requests 2 Projects Releases Packages Wiki Activity Actions

fix(asus): own the 8TB backup mounts in ansible and alert on stale backups #186

Merged
kblack0610 merged 1 commit from fix/asus-backup-mount-durability into master 2026-08-13 02:35:11 +00:00
Conversation 0 Commits 1 Files changed 7 +283 -4
kblack0610 commented 2026-08-13 02:34:59 +00:00
Owner
Copy link

Follow-up to #177. That PR made the backup jobs fail loudly when the drive is gone; this one stops the drive from going gone in the first place, and notices within the hour if a backup stops happening for any other reason.

1. roles/usb-backup-mounts

The seven btrfs subvolume mounts move to noauto,x-systemd.automount. systemd-fstab-generator binds each .mount unit to the underlying .device unit:

Event Before After
Device disappears mount goes stale, still listed, mountpoint still true systemd stops the .mount unit; path left clean
Device returns (any node name) nothing - stays broken until noticed next access remounts, resolved by UUID

That is the whole 2026-08-09 failure: the drive came back as /dev/sda instead of /dev/sdc, nothing remounted, and df/mountpoint/node_filesystem_* all kept reporting a healthy 7.3T/177G over a device node that no longer existed.

2. Inventory: asus-laptop loses unreachable: true

It was stale - the host answers fine on port 2222 with the standard key - and it was load-bearing in the wrong direction: because the host was marked unmanaged, its backup drive had no owning role, so when the mounts broke there was nothing to fix them with and they had to be repaired by hand.

3. prometheus-rules-backup-drive.yaml

HomelabBackupStale (daily, >36h) and HomelabWeeklyBackupStale (nas namespace, >9d).

Deliberately measures the outcome, not the plumbing: during the incident every plumbing signal was green and the only false statement was "a backup completed recently". An absent()-on-mountpoint alert was considered and rejected - with automount a path is legitimately unmounted until first access, so it would fire after every reboot. Daily/weekly are split because nas/ holds only weekly jobs and one flat threshold would page on all four every week for being healthy (the same mistake caught in #173).

Verification (live host + cluster)

  • --check --diff first, then applied: ok=8 changed=1 failed=0
  • All 7 automount units enabled + active; accessing a path remounts it; real source underneath is /dev/sda1 with 0 read/write/corruption errors
  • Data intact: 14 immich snapshots, 25 db dumps, 29 3d-print entries
  • Full backup run through the automount path: rsync + db dump + new snapshot + prune, succeeded=1
  • Both alert expressions run live: the daily rule correctly flags immich-originals-backup at 88h stale; the weekly rule correctly stays silent on the four nas jobs at 3.7d

Transition note: on a host where the paths are already mounted, the automount unit cannot establish over them, so the switchover needs a one-time unmount (or a reboot). Done here; documented in the role README.

Follow-up to #177. That PR made the backup jobs **fail loudly** when the drive is gone; this one stops the drive from going gone in the first place, and notices within the hour if a backup stops happening for any other reason. ## 1. `roles/usb-backup-mounts` The seven btrfs subvolume mounts move to `noauto,x-systemd.automount`. `systemd-fstab-generator` binds each `.mount` unit to the underlying `.device` unit: | Event | Before | After | |---|---|---| | Device disappears | mount goes stale, still listed, `mountpoint` still true | systemd stops the `.mount` unit; path left clean | | Device returns (any node name) | nothing - stays broken until noticed | next access remounts, resolved by UUID | That is the whole 2026-08-09 failure: the drive came back as `/dev/sda` instead of `/dev/sdc`, nothing remounted, and `df`/`mountpoint`/`node_filesystem_*` all kept reporting a healthy 7.3T/177G over a device node that no longer existed. ## 2. Inventory: asus-laptop loses `unreachable: true` It was stale - the host answers fine on port 2222 with the standard key - and it was load-bearing **in the wrong direction**: because the host was marked unmanaged, its backup drive had no owning role, so when the mounts broke there was nothing to fix them with and they had to be repaired by hand. ## 3. `prometheus-rules-backup-drive.yaml` `HomelabBackupStale` (daily, >36h) and `HomelabWeeklyBackupStale` (nas namespace, >9d). Deliberately measures the **outcome**, not the plumbing: during the incident every plumbing signal was green and the only false statement was *"a backup completed recently"*. An `absent()`-on-mountpoint alert was considered and **rejected** - with automount a path is legitimately unmounted until first access, so it would fire after every reboot. Daily/weekly are split because `nas/` holds only weekly jobs and one flat threshold would page on all four every week for being healthy (the same mistake caught in #173). ## Verification (live host + cluster) - `--check --diff` first, then applied: `ok=8 changed=1 failed=0` - All 7 automount units enabled + active; accessing a path remounts it; real source underneath is `/dev/sda1` with **0** read/write/corruption errors - Data intact: **14** immich snapshots, **25** db dumps, **29** 3d-print entries - Full backup run through the automount path: rsync + db dump + new snapshot + prune, `succeeded=1` - Both alert expressions run live: the daily rule correctly flags `immich-originals-backup` at **88h** stale; the weekly rule correctly stays **silent** on the four nas jobs at 3.7d > **Transition note:** on a host where the paths are already mounted, the automount unit cannot establish over them, so the switchover needs a one-time unmount (or a reboot). Done here; documented in the role README.
Follow-up to #177. That PR made the backup jobs FAIL loudly when the drive
is gone; this one stops the drive from going gone in the first place, and
notices within the hour if a backup stops happening for any other reason.

Three parts.

1. roles/usb-backup-mounts - the seven btrfs subvolume mounts move to
   noauto,x-systemd.automount. systemd-fstab-generator then binds each
   .mount unit to the underlying .device unit, so device loss tears the
   mount down cleanly instead of leaving a stale entry, and device return
   remounts by UUID on next access. That is the whole 2026-08-09 failure:
   the drive came back as /dev/sda instead of /dev/sdc, nothing remounted,
   and df/mountpoint/node_filesystem all kept reporting a healthy
   7.3T/177G over a device node that no longer existed.

2. inventory: asus-laptop loses `unreachable: true`. It was stale - the
   host answers fine on port 2222 with the standard key - and it was
   load-bearing in the wrong direction: because the host was marked
   unmanaged, its backup drive had no owning role, so when the mounts broke
   there was nothing to fix them with and they had to be repaired by hand.

3. prometheus-rules-backup-drive.yaml - HomelabBackupStale (daily, >36h)
   and HomelabWeeklyBackupStale (nas namespace, >9d). Deliberately measures
   the OUTCOME, not the plumbing: during the incident every plumbing signal
   was green and the only false statement was "a backup completed
   recently". An absent()-on-mountpoint alert was considered and rejected -
   with automount a path is legitimately unmounted until first access, so
   that alert would fire after every reboot. Split daily from weekly
   because nas/ holds only weekly jobs and one flat threshold would page on
   all four every week for being healthy.

Verified against the live host and cluster:
  - --check --diff first, then applied: ok=8 changed=1 failed=0
  - all 7 automount units enabled + active; accessing a path remounts it,
    real source underneath is /dev/sda1, 0 read/write/corruption errors
  - data intact: 14 immich snapshots, 25 db dumps, 29 3d-print entries
  - full backup run through the automount path: rsync + db dump + new
    snapshot + prune, succeeded=1
  - both alert expressions run live: the daily rule correctly flags
    immich-originals-backup at 88h stale, the weekly rule correctly stays
    silent on the four nas jobs at 3.7d

Transition note: on a host where the paths are already mounted the
automount unit cannot establish over them, so the switchover needs a
one-time unmount (or a reboot). Done here; documented in the role README.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kblack0610/home-config!186
No description provided.