Skip to content

scheduled-maintenance — hardening items run unattended, reports left for review

Deterministic bash under root systemd one-shot timers, no Claude in the loop. Each job refuses to start if a sync, scrub or backup is running, writes a Markdown report to ~/maintenance/output/, and pushes ntfy. Folder contract: ~/maintenance/CONTEXT.md.

Input → Movement → Output

Reads the array. maint-badblocks reads 20 GB of disk1 (item 17, read-only). maint-drill syncs, audits d3, then moves one 1 MB file out and proves snapraid fix reconstructs it byte-for-byte (item 15). maint-dedupe hashes every same-disk size-matched download↔library pair and, if every check passes, replaces the download copy with a hardlink (item 16 B). Output: reports + .log/.tsv evidence.

Why this shape

These items are read-heavy, hours long, and need nothing a script cannot decide — except the dedupe link step, which the owner chose to auto-run on a clean scan. Doing them as cron-style jobs costs zero tokens and survives a dropped SSH session; a Claude session only reads the result.

Steps

  1. ~/maintenance/install.sh (root, once): installs /usr/local/bin/maint-*.sh, /etc/systemd/system/maint-*.{service,timer}, /etc/logrotate.d/docker-backup; revokes /etc/sudoers.d/90-claude-maintenance.
  2. Guard in maint-common.sh:busy_guard — mounts, snapraid-sync/scrub inactive, no snapraid/badblocks/backup process.
  3. maint-badblocks.sh: badblocks -sv -b 4096 -o … <uuid-dev> 704700000 699900000 (last block first) → report with kernel-error count and SMART reallocation attrs.
  4. maint-drill.sh: stop timers → snapraid-sync-guard.sh → snapraid -d d3 -f <rel> list must match 1 → snapraid -d d3 -a check → mv file out → -m check dry run → -m fix → cmp → restart timers (trap restores file + timers on any failure).
  5. maint-dedupe.sh: same-disk, >100 MiB, different inode, library not .unrecoverable → sha256 both (cached per inode) → ln -f LIB DL.tmp && mv -f DL.tmp DL → verify inodes equal. AUTO_LINK=0 makes it report-only.
  6. Timers are dated (OnCalendar=YYYY-MM-DD HH:MM, Persistent=false): a missed run is rescheduled, never fired at boot.

Incident 2026-09-08 23:00 — false "SnapRAID FAILED" alert

maint-badblocks refused to start: the busy-guard's pgrep -f matched the job's own $(…) subshell (same cmdline, PID ≠ $$), and OnFailure=snapraid-alert@ then pushed it under a SnapRAID title. SnapRAID itself was fine — the 03:30 sync ran clean and absorbed the 90 G history repo. Fixed 09-09 06:00: the guard ignores PIDs in its own cgroup and dead PIDs (tested both ways); maint units now fail to maint-alert@ ("Maintenance job FAILED: … (not SnapRAID)"); install.sh re-dates missed timers. Re-run the installer after any script change — /usr/local/bin copies are root-owned.

Finding 2026-09-09 — the precheck wakes the parity disk and hd-idle never notices

During the drill, sdb (parity) climbed 36→55 °C over 2.5 h while doing zero block I/O. Cause: snapraid-precheck.sh ran smartctl -H/-A /dev/sdb, which spins a sleeping drive up; hd-idle only watches /proc/diskstats, saw no I/O, and believed the disk was still stopped — so it never issued another spin-down. The 08:00 sync was a no-op ("Nothing to do"), so nothing else touched sdb until 10:27; hd-idle then counted 1801 s and spun it down at 10:57. The same thing happens on any night the 03:30 sync has nothing to do. Fixed 09-09 11:05: the precheck now calls smartctl -n standby and skips a sleeping drive (smartd, which already uses -n standby, covers it). Backup ~/snapraid-precheck.sh.bak.20260909-parity-standby. Item 14 (intake fan) remains the structural answer — the stack ran 4 °C hotter than the day before from one disk being read.

If you change this

  • Hits: the hardening-plan card statuses (close 15/16/17 from the reports) · tonight's 03:30 sync after dedupe (N updated < 2,000) · snapraid diff counts.
  • Does not hit: parity contents (drill's fix writes one file; dedupe changes inodes, not blocks) · containers (qBittorrent keeps seeding identical bytes).

Surfaces

Surface Role
systemctl list-timers 'maint-*' what is still pending
~/maintenance/output/ reports; maint-<job>.journal.log for stdout
ntfy plex-hgviaey completion / needs-review

See