fix(backups): detect a stale mount, not just a mounted path #177

Merged

kblack0610 merged 1 commit from fix/stale-mount-backup-guard into master

2026-08-11 22:09:16 +00:00

kblack0610 commented

2026-08-11 22:09:06 +00:00

Owner

The 8TB backup drive is unplugged

Aug 09 09:39:43 asus-laptop kernel: usb 4-2: USB disconnect, device number 3
Aug 09 09:39:43 asus-laptop kernel: sd 2:0:0:0: [sdc] Synchronizing SCSI cache
Aug 09 09:39:43 asus-laptop kernel: sd 2:0:0:0: [sdc] Synchronize Cache(10) failed

/dev/sdc has not been a block device since. Nothing has been backed up to it for two days. 59k+ btrfs read errors and counting; /backup-dest reads as empty and all 14 immich snapshots throw I/O errors.

The guard written for this exact scenario did not catch it

When a USB device disappears the filesystem stays mounted: mountpoint -q returns true, df still reports 7.3T/177G used, and only real I/O fails with EIO. So the guard waved every run through, rsync failed mid-transfer, and the job reported BackoffLimitExceeded with no explanation.

Verified against the live stale mount

A rare chance to test a guard against the precise condition it exists for:

	result
old guard (`mountpoint` only) on the dead mount	PASSES - job proceeds, backs up nothing
new guard on the dead mount	aborts: `backing device is gone (stale mount)`
new guard on a healthy mount (control)	passes

Note on the probe: it uses touch, not : > file. : is a POSIX special builtin, so a redirection failure makes the shell exit immediately rather than return non-zero - the abort message would never print and the exit would be accidental rather than designed. Caught by testing, not by reading.

Both consumers of the drive (immich/originals-backup, nas/3d-prints-backup) get the same guard, and both move to restartPolicy: Never - OnFailure makes the Job controller delete the failed pod and take its logs with it, which is why two days of failures could only be seen as "no pods retained".

This does not restore the backup. The drive needs physically reconnecting. Until then each run now fails loudly and legibly instead of dying six minutes into an rsync with no message.

## The 8TB backup drive is unplugged ``` Aug 09 09:39:43 asus-laptop kernel: usb 4-2: USB disconnect, device number 3 Aug 09 09:39:43 asus-laptop kernel: sd 2:0:0:0: [sdc] Synchronizing SCSI cache Aug 09 09:39:43 asus-laptop kernel: sd 2:0:0:0: [sdc] Synchronize Cache(10) failed ``` `/dev/sdc` has not been a block device since. **Nothing has been backed up to it for two days.** 59k+ btrfs read errors and counting; `/backup-dest` reads as empty and all 14 immich snapshots throw I/O errors. ## The guard written for this exact scenario did not catch it When a USB device disappears the filesystem stays **mounted**: `mountpoint -q` returns true, `df` still reports 7.3T/177G used, and only real I/O fails with EIO. So the guard waved every run through, rsync failed mid-transfer, and the job reported `BackoffLimitExceeded` with no explanation. ## Verified against the live stale mount A rare chance to test a guard against the precise condition it exists for: | | result | |---|---| | old guard (`mountpoint` only) on the dead mount | **PASSES** - job proceeds, backs up nothing | | new guard on the dead mount | aborts: `backing device is gone (stale mount)` | | new guard on a healthy mount (control) | passes | **Note on the probe:** it uses `touch`, not `: > file`. `:` is a POSIX *special builtin*, so a redirection failure makes the shell exit immediately rather than return non-zero - the abort message would never print and the exit would be accidental rather than designed. Caught by testing, not by reading. Both consumers of the drive (`immich/originals-backup`, `nas/3d-prints-backup`) get the same guard, and both move to `restartPolicy: Never` - `OnFailure` makes the Job controller delete the failed pod and take its logs with it, which is why two days of failures could only be seen as "no pods retained". > This does **not** restore the backup. The drive needs physically reconnecting. Until then each run now fails loudly and legibly instead of dying six minutes into an rsync with no message.

kblack0610 added 1 commit

2026-08-11 22:09:06 +00:00

fix(backups): detect a stale mount, not just a mounted path 198c1bda68

The 8TB USB backup drive was physically disconnected on 2026-08-09
09:39:43 (`usb 4-2: USB disconnect`, followed by a failed SCSI cache
sync). /dev/sdc has not been a block device since. Nothing has been backed
up to it for two days.

The mount-safety guard was written for exactly this scenario and did not
catch it. When a USB device disappears the filesystem stays MOUNTED:
`mountpoint -q` still returns true, `df` still reports 7.3T with 177G
used, and only real I/O fails with EIO. So the guard waved every run
through, rsync then failed mid-transfer, and the job reported
BackoffLimitExceeded with no explanation.

Verified against the live stale mount, which is a rare chance to test a
guard against the precise condition it exists for:
  old guard (mountpoint only) -> PASSES on the dead mount
  new guard                   -> aborts with "backing device is gone"
  new guard on a healthy mount -> passes (control)

Note on the probe: it uses `touch`, not `: > file`. `:` is a POSIX special
builtin, so a redirection failure makes the shell EXIT immediately rather
than return non-zero - the abort message would never print and the exit
would be accidental rather than designed. Caught by testing.

Both consumers of the drive get the same guard. Both also move to
restartPolicy: Never, because OnFailure makes the Job controller delete the
failed pod and take its logs with it - which is why two days of failures
could only be seen as "no pods retained" and had to be diagnosed by hand.

This does NOT restore the backup. The drive needs physically reconnecting;
until then the new guard makes each run fail loudly and legibly instead of
failing six minutes into an rsync with no message.