Skip to content

Alerting — where every alarm goes, and what stays silent

One ntfy topic for failures (plex-hgviaey, phone push) and Uptime Kuma push monitors for heartbeats. Before 2026-09-02 nothing on the box could reach a human: smartd mailed root through a mailer that exits 1 (no MTA), Scrutiny had no config file, and 30 consecutive sync failures in August went unseen.

Shape

Source Trigger Path Tested
Scrutiny any drive status change ~/torrentvpn/scrutiny/config/scrutiny.yaml → ntfy://ntfy.sh/plex-hgviaey 09-02 push
smartd health/attribute/self-test failure, temp thresholds (-W) -M exec /usr/local/bin/smartd-ntfy.sh (bounded, 20 s) 09-03 -M test push
snapraid-sync / snapraid-scrub units unit failure (precheck abort counts) OnFailure=snapraid-alert@%n.service → last 25 journal lines to ntfy 09-02 selftest
rescue-watchdog archive > 30 h, < 100 MB, stick missing or wrong device NTFY_URL + WEBHOOK_URL from /etc/rescue-kit.conf fires at boot after a power-off (expected)
rescue-sync build success KUMA_PUSH_URL heartbeat + DEADMAN_URL nightly "heartbeat sent"
precious-sync copy success PRECIOUS_KUMA_PUSH_URL heartbeat nightly
history-backup any failure, prune > 400 packs, weekly drill FAIL, initial-run completion ntfy; optional HISTORY_KUMA_PUSH_URL heartbeat 09-08 initial run
vpn-check gluetun egress on the home ISP (Brightspeed/AS19901) or down homer tile red; vpn-status.json every 5 min
tasks-digest run failed, or did not run at all OnFailure=tasks-digest-alert → ntfy; a 09:30 canary compares the heartbeat file's date to today 09-13 both directions proven
surfaces-check an internal page is ungated, blank, has a script that does not parse, or its API is empty ntfy; twice daily 06:40 and 15:40 09-13 proven by breaking home.js on purpose
uptime-kuma any monitor down whatever is configured in Kuma (uptime-kuma container) —

Silent by design or by omission (know this): - rsnapshot — no alerting; check /mnt/disk1/Storage-snapshots/ mtimes. - docker-backup.sh — logs only; rescue-sync would fail downstream if a snapshot were missing. - training-sync — log + last-run-<mode>.json; no push on failure. A usage-limit rejection makes it sleep until the reset and retry until the mode's deadline. - hd-idle, wg watchdogs — journal only. - A precheck abort at 03:30 is a failure of the sync unit and therefore pages. Good — but it also means the array is unsynced until someone reads the alert.

Connected to

If you change this

  • Hits: the ntfy topic string appears in four places: scrutiny.yaml, smartd-ntfy.sh:12, snapraid-alert@.service, /etc/rescue-kit.conf NTFY_URL — change all four.
  • Does not hit: the Kuma heartbeats (separate URLs, in rescue-kit.conf).

See

  • Source: /usr/local/bin/smartd-ntfy.sh · /etc/systemd/system/snapraid-alert@.service · /usr/local/bin/rescue-watchdog.sh:46-59 · ~/.local/bin/vpn-check