Rescue USB — the bare-metal rebuild kit for a dead OS disk¶
A 14.4 GB stick at /mnt/rescue holding the configuration (not media): nightly container-config archives, the encrypted .env, and the host files a fresh install cannot infer. Rebuilt every night; watched every morning.
Why this shape¶
The array holds 5 TB of media that survives an OS-disk death on its own. What does not survive is years of container configuration, fstab, snapraid.conf and — critically — snapraid.content, without which the parity disk is an undifferentiated blob. Those fit on a stick, so they live on one.
Shape¶
On the stick: latest.tar.zst → newest configs-YYYY-MM-DD.tar.zst (up to 14 kept; the oldest are dropped until the build fits, never below 3; ~900 MB each as of 2026-10-10, .bak copies excluded; warns at >25% weekly growth or <7 nights) · env.gpg (passphrase in the password manager, never on the stick) · host/ (fstab, snapraid.conf, snapraid.content*, sync-guard + precheck, systemd units, cron.d, crontabs, local-bin/, systemd-user/, sshd hardening, sysctl) · restore.sh · MANIFEST.sha256 · README.txt (the rebuild order) · .rescue-usb-id (sentinel = filesystem UUID).
Source directory: ~/rescue-kit/ (install.sh installs to /usr/local/bin and /etc/rescue-kit.conf; format-usb.sh; install-snapraid-alerts.sh). Operator CLI: rk (rk status · rk run · rk test restore-drill · rk log -f · rk set K V). Config keys set in /etc/rescue-kit.conf: GPG_PASSFILE KUMA_PUSH_URL DEADMAN_URL NTFY_URL WEBHOOK_URL PRECIOUS_KUMA_PUSH_URL.
Rebuild order lives in README.txt on the stick; the steps that bite: restore the guard + precheck before enabling the sync timer; put snapraid.content on all three data disks before the first sync; add your SSH key before restarting sshd (99-hardening.conf disables passwords); gluetun needs HEALTH_SMALL_CHECK_TYPE=dns; recreate gluetun and qbittorrent together.
What it does NOT cover — 2026-09-25¶
Two limits are easy to miss, because host/local-bin/ on the stick makes ~/.local/bin look
fully protected. It is protected against losing the NVMe, and against nothing else.
- No history. Each run overwrites
host/with today's copy. Break a script and not notice for four days and every archive on the stick has the broken version. The 14 keptconfigs-*.tar.zstare the docker snapshot;host/is a single rolling copy beside them. - On-site only. The stick sits in this building, in the same chassis' USB port. One fire or
theft takes the server and the backup together.
erp-backuppushes to R2 off-box; the rescue stick does not.
Source repos are outside its scope entirely. It captures /etc config, snapraid.content,
systemd units, crontabs and the docker snapshot — not code written in ~. Verified 2026-09-25 by
searching the whole stick: ~/ee-api and ~/embeddedera-store appear nowhere on it.
| Path | Off-box copy | History |
|---|---|---|
~/Notes |
git remote | yes |
~/torrentvpn · ~/kicad-lib · ~/assistant |
GitHub EmbeddedEra/* |
yes |
~/ee-api |
GitHub EmbeddedEra/ee-api (created 2026-09-25) |
yes |
~/embeddedera-store |
none — blocked on a gh repo create permission denial |
no |
~/.local/bin |
rescue stick only (on-site) | no |
Read this card before proposing a backup for anything on this box. On 2026-09-25 three new
GitHub repos were proposed as disaster recovery for code that host/local-bin/ already carried;
the list in the contents line above answers that in one read.
Connected to¶
- owned-by: disks (sde,
nofail) - joins: processes/rescue-build (builds it), automation/alerting (watchdog 09:00)
- looks-like-but-is-not:
/mnt/storage/docker-backups/— the same configs, on the array, withoutcompose/or.env.
If you change this¶
- Hits:
rescue-sync.shrefuses a stick whose UUID ≠ sentinel (reformat ⇒rk setup) · the fstabnofailline ·rescue-watchdog(alerts if archive > 30 h old or < 100 MB, or wrong device) · after a power-off, the watchdog'sPersistent=truefires once at boot — an alert right after power-on is expected unless the archive is genuinely stale. - Does not hit: the array, parity, containers.
Surfaces¶
| Surface | Role |
|---|---|
| root crontab | 01:30 nightly build (after docker-backup.sh at 01:00) |
rescue-watchdog.timer |
09:00 + ≤5 min jitter, /var/log/rescue-watchdog.log |
| quarterly drill | sha256sum -c MANIFEST.sha256; rk test — a backup never restored is a hypothesis |
See¶
- Source:
/usr/local/bin/rescue-sync.sh·/usr/local/bin/rescue-watchdog.sh·/mnt/rescue/README.txt(copy at~/rescue-kit/README.txt)