fix(forgejo): cap backup retention at 7 (cluster-wide DiskPressure outage fix) #108

Merged

kblack0610 merged 1 commit from fix/forgejo-backup-retention into master

2026-07-09 17:02:24 +00:00

kblack0610 commented

2026-07-09 17:02:13 +00:00

Owner

Root cause of the 2026-07-09 ~6h home-k3s outage: forgejo-backup retention=30 x ~55G fulls filled asus-laptop root disk -> DiskPressure -> whole workload cluster evicted. Cuts retention to 7, prunes before writing, and aborts if <100G free. Old backups already pruned on-host to restore service.

kblack0610 added 1 commit

2026-07-09 17:02:14 +00:00

fix(forgejo): cap backup retention at 7, prune-first + free-space guard 2bf63f1aaa

The forgejo-backup CronJob wrote a full ~55G tarball nightly to hostPath
/var/backups/forgejo on asus-laptop's root disk with retention=30. Steady
state ~1.6TiB filled the 1.9TB disk on 2026-07-09, tripped kubelet
DiskPressure, tainted the node, and evicted the entire home-k3s workload
cluster (forgejo, immich, minio, postgres, monitoring, ...) for ~6h. git
push + Flux reconcile were down because forgejo could not schedule.

Changes:
- Retention 30 -> 7 full backups (~55G each; ~385G steady state).
- Prune BEFORE writing so the disk never holds more than KEEP backups.
- Abort with a loud error if /backup has < 100G free, instead of filling
  the node and taking it down again.
- set -eu; document the incident in the README.

Follow-up (not in this commit): move long-term recovery points off-box
(MinIO/S3) so backups don't share a disk with the data they protect, and
add a filesystem-usage / DiskPressure alert that pages before it fills.