Skip to content

rescue-build — snapshot the configs, archive them to the USB, prove the stick is alive

Three movements in sequence each night: docker-backup.sh rsyncs every container config dir to the array (01:00); rescue-sync.sh tars the newest snapshot onto the rescue stick with the host files and pushes a heartbeat (01:30); rescue-watchdog checks the stick the next morning (09:00) independently of both.

Input → Movement → Output

Reads ~/torrentvpn/*/ (minus caches, logs, Plex metadata, media dirs) and the host config set. Produces a 7-deep snapshot tree on the pool, a 14-deep archive set on the stick with MANIFEST.sha256, and a fresh host/ (fstab, snapraid.conf, snapraid.content, guard + precheck, units, crontabs, local-bin/, systemd-user/).

Why this shape

Seven months of backups once went to the SSD because a mount was missing; both scripts now refuse to write unless the destination is a real mountpoint (docker-backup.sh:11-19, rescue-sync.sh:37) and the stick's UUID matches its sentinel (rescue-sync.sh:44-45). The watchdog is separate so a bug in the builder cannot mask a stopped builder.

Steps

  1. 01:00 root cron → /usr/local/bin/docker-backup.sh: snapshot compose/ + configs/ under /mnt/storage/docker-backups/snapshots/<ts>/; keep 7. Log /var/log/docker-backup.log (832 MB, rotate it).
  2. 01:30 root cron → /usr/local/bin/rescue-sync.sh: guards → (re-runs docker-backup if the snapshot is stale, :71) → archive .tmp then rename → host/ collection (:100-112) → MANIFEST.sha256 → Kuma heartbeat + deadman. Keeps 14; 2 GiB free floor. Log /var/log/rescue-sync.log.
  3. 09:00 rescue-watchdog.timer → checks mount, archive age ≤ 30 h, size ≥ 100 MB, UUID = sentinel; ntfy/webhook on any problem (rescue-watchdog.sh:24-41).
  4. Quarterly by hand: rk test (restore to scratch + sqlite integrity check); sha256sum -c MANIFEST.sha256.

If you change this

  • Hits: rk (reads the same conf) · precious-sync.sh shares /etc/rescue-kit.conf but its own key · the snapshot dir is exclude /docker-backups/ in snapraid.conf — keep it that way (493 parity errors a night otherwise) · docker-backup.sh:49 skip-list if you add a data-heavy dir to the stack.
  • Does not hit: media, parity, the precious copy.

Surfaces

Surface Role
rk status / run / test / log
Uptime Kuma + ntfy heartbeat / failure

See

  • Objects: rescue-usb, torrentvpn-stack
  • Source: /usr/local/bin/docker-backup.sh · /usr/local/bin/rescue-sync.sh · /usr/local/bin/rescue-watchdog.sh · ~/rescue-kit/README.txt