Skip to content

Rescue USB — the bare-metal rebuild kit for a dead OS disk

A 14.4 GB stick at /mnt/rescue holding the configuration (not media): nightly container-config archives, the encrypted .env, and the host files a fresh install cannot infer. Rebuilt every night; watched every morning.

Why this shape

The array holds 5 TB of media that survives an OS-disk death on its own. What does not survive is years of container configuration, fstab, snapraid.conf and — critically — snapraid.content, without which the parity disk is an undifferentiated blob. Those fit on a stick, so they live on one.

Shape

On the stick: latest.tar.zst → newest configs-YYYY-MM-DD.tar.zst (14 kept, ~389 MB) · env.gpg (passphrase in the password manager, never on the stick) · host/ (fstab, snapraid.conf, snapraid.content, sync-guard + precheck, systemd units, cron.d, crontabs, local-bin/, systemd-user/, sshd hardening, sysctl) · restore.sh · MANIFEST.sha256 · README.txt (the rebuild order) · .rescue-usb-id (sentinel = filesystem UUID).

Source directory: ~/rescue-kit/ (install.sh installs to /usr/local/bin and /etc/rescue-kit.conf; format-usb.sh; install-snapraid-alerts.sh). Operator CLI: rk (rk status · rk run · rk test restore-drill · rk log -f · rk set K V). Config keys set in /etc/rescue-kit.conf: GPG_PASSFILE KUMA_PUSH_URL DEADMAN_URL NTFY_URL WEBHOOK_URL PRECIOUS_KUMA_PUSH_URL.

Rebuild order lives in README.txt on the stick; the steps that bite: restore the guard + precheck before enabling the sync timer; put snapraid.content on all three data disks before the first sync; add your SSH key before restarting sshd (99-hardening.conf disables passwords); gluetun needs HEALTH_SMALL_CHECK_TYPE=dns; recreate gluetun and qbittorrent together.

Connected to

If you change this

  • Hits: rescue-sync.sh refuses a stick whose UUID ≠ sentinel (reformat ⇒ rk setup) · the fstab nofail line · rescue-watchdog (alerts if archive > 30 h old or < 100 MB, or wrong device) · after a power-off, the watchdog's Persistent=true fires once at boot — an alert right after power-on is expected unless the archive is genuinely stale.
  • Does not hit: the array, parity, containers.

Surfaces

Surface Role
root crontab 01:30 nightly build (after docker-backup.sh at 01:00)
rescue-watchdog.timer 09:00 + ≤5 min jitter, /var/log/rescue-watchdog.log
quarterly drill sha256sum -c MANIFEST.sha256; rk test — a backup never restored is a hypothesis

See

  • Source: /usr/local/bin/rescue-sync.sh · /usr/local/bin/rescue-watchdog.sh · /mnt/rescue/README.txt (copy at ~/rescue-kit/README.txt)