fix(backups): stop the remaining jobs reporting a NAS copy they did not make #205

Merged

kblack0610 merged 1 commit from fix/backup-smbclient-checked into master

2026-08-26 23:28:15 +00:00

kblack0610 commented

2026-08-26 23:27:46 +00:00

Owner

Finishes the sweep started in #202, as requested.

The ask

home-assistant, immich, litellm, vaultwarden and the actual-budget tax-export all still used the unchecked form:

smbclient ... -c "mkdir a; cd a; put ..." && echo "NAS copy OK" || echo "WARN"

smbclient -c does not exit non-zero when a command inside the string fails, so every one of these would report a successful off-box copy for a file that had gone to the share root or nowhere. All five now use the checked smb() wrapper from apps/zomboid/backup/backup.sh, build the destination one component at a time, and confirm the file is in the right directory at the right size.

Off-box policy is unchanged for these five - the local copy is still the success condition, a NAS outage still warns and exits 0. actual-budget stays the only job that pages (#203), per the decision on that PR.

Guards added where they cannot false-trip

Job	Floor	Current actual
home-assistant	1MB	~250MB
immich	1MB	~150MB
vaultwarden	4KB	~40KB

immich also gets set -o pipefail. Without it a failed pg_dump produced a perfectly valid gzip of nothing and reported success. I verified alpine's ash actually supports pipefail before relying on it, since a bare set -o pipefail under set -e would otherwise have killed the job outright.

litellm: found a real gap, deliberately not papered over

litellm-backup has never backed up anything. Every archive it has written is 104 bytes. The job tars the litellm-data PVC, but that PVC is empty - litellm keeps its virtual keys, spend ledger and config in Postgres via DATABASE_URL. The result is a valid, well-formed backup of an empty directory, shipped to the NAS nightly, reporting success.

It gets no size floor on purpose: a nightly page would announce the gap without closing it, and the job needs repointing at the database, not an alarm. Recorded under Known gaps in the runbook with commands to verify it, and flagged as needing a decision - it is the same missing piece as the shared Postgres having no backup at all (mem0 included).

Verification

Each script run in a container with a stubbed smbclient, against real data:

Case	Result
happy path	exit 0, uploads, size-verified
`cd` fails, lands in share root	WARN, never "NAS copy OK"
archive under floor	exit 1 with the reason
`pg_dump` fails (immich)	exit 1, no empty dump shipped

Finishes the sweep started in #202, as requested. ## The ask `home-assistant`, `immich`, `litellm`, `vaultwarden` and the `actual-budget` tax-export all still used the unchecked form: ```sh smbclient ... -c "mkdir a; cd a; put ..." && echo "NAS copy OK" || echo "WARN" ``` `smbclient -c` does not exit non-zero when a command inside the string fails, so every one of these would report a successful off-box copy for a file that had gone to the share root or nowhere. All five now use the checked `smb()` wrapper from `apps/zomboid/backup/backup.sh`, build the destination one component at a time, and confirm the file is in the right directory at the right size. **Off-box policy is unchanged for these five** - the local copy is still the success condition, a NAS outage still warns and exits 0. `actual-budget` stays the only job that pages (#203), per the decision on that PR. ## Guards added where they cannot false-trip | Job | Floor | Current actual | |---|---|---| | home-assistant | 1MB | ~250MB | | immich | 1MB | ~150MB | | vaultwarden | 4KB | ~40KB | `immich` also gets `set -o pipefail`. Without it a failed `pg_dump` produced a perfectly valid gzip of nothing and reported success. I verified alpine's ash actually supports `pipefail` before relying on it, since a bare `set -o pipefail` under `set -e` would otherwise have killed the job outright. ## litellm: found a real gap, deliberately not papered over **`litellm-backup` has never backed up anything.** Every archive it has written is 104 bytes. The job tars the `litellm-data` PVC, but that PVC is empty - litellm keeps its virtual keys, spend ledger and config in Postgres via `DATABASE_URL`. The result is a valid, well-formed backup of an empty directory, shipped to the NAS nightly, reporting success. It gets **no size floor** on purpose: a nightly page would announce the gap without closing it, and the job needs repointing at the database, not an alarm. Recorded under *Known gaps* in the runbook with commands to verify it, and flagged as needing a decision - it is the same missing piece as the shared Postgres having no backup at all (mem0 included). ## Verification Each script run in a container with a stubbed smbclient, against real data: | Case | Result | |---|---| | happy path | exit 0, uploads, size-verified | | `cd` fails, lands in share root | WARN, never "NAS copy OK" | | archive under floor | exit 1 with the reason | | `pg_dump` fails (immich) | exit 1, no empty dump shipped |

kblack0610 added 1 commit

2026-08-26 23:27:46 +00:00

fix(backups): stop the remaining jobs reporting a NAS copy they did not make

checks / fleet-roster (pull_request) Successful in 6s

Details

checks / kustomize-builds (pull_request) Successful in 7s

Details

f9933b411e

Finishes the sweep started in #202. home-assistant, immich, litellm,
vaultwarden and the actual-budget tax-export all still used the unchecked
`smbclient -c "...; ...; put ..." && echo "NAS copy OK"` form. smbclient does
not exit non-zero when a command inside the string fails, so each of them
would report a successful off-box copy for a file that had landed in the
share root or nowhere. All five now use the checked smb() wrapper from
apps/zomboid/backup/backup.sh, create the destination one component at a
time, and confirm the uploaded file is in the right directory at the right
size.

Off-box policy is unchanged for these five: the local copy is still the
success condition and a NAS outage still warns and exits 0. actual-budget
remains the only job that pages (#203).

Also adds refuse-to-back-up-nothing floors where they are protective and
cannot false-trip: home-assistant (1MB against ~250MB actual), vaultwarden
(4KB against ~40KB), immich (1MB against ~150MB). immich additionally gets
`set -o pipefail`, without which a failed pg_dump produced a perfectly valid
gzip of nothing and reported success - verified that alpine's ash supports
pipefail before relying on it.

litellm deliberately gets NO floor. Its archives are 104 bytes and always
have been: the litellm-data PVC is empty because litellm keeps its keys,
spend and config in Postgres. A floor would page nightly for a gap that
needs the job repointed at the database, not an alarm. Documented as a
known gap in the runbook instead, with commands to verify it.

Verified each script with a stubbed smbclient against real data: happy path
uploads and size-verifies; a failing `cd` produces WARN and never
"NAS copy OK"; undersized archives exit 1 with the reason; a failing
pg_dump exits 1 instead of shipping an empty dump.