Alerting — where every alarm goes, and what stays silent

One ntfy topic for failures (plex-hgviaey, phone push) and Uptime Kuma push monitors for heartbeats. Before 2026-09-02 nothing on the box could reach a human: smartd mailed root through a mailer that exits 1 (no MTA), Scrutiny had no config file, and 30 consecutive sync failures in August went unseen.

Shape

Source	Trigger	Path	Tested
codex-auth-heal	Codex's ChatGPT login dies (refresh token expired/invalidated; ai-capacity refresh fails and a codex exec probe shows 401/refresh-token)	user timer every 15 min → ~/.local/bin/codex-auth-heal: starts codex login --device-auth, pushes link + code to ntfy (tap-to-open); after approval restarts the app-server daemon and reset-faileds ai-capacity-refresh / codex-remote-control / codex-update	2026-10-18 healthy run silent; dead-login path proven with an empty CODEX_HOME (one login started, no duplicate); first real outage not yet seen
ai-budget watch	a Claude, Codex or API pool runs >10 points ahead of pace (once per pool per window)	ai-capacity-refresh.service.d/ai-budget.conf → ntfy.sh/plex-hgviaey	10-16 live push
UPS (NUT upsmon)	on battery, back online, low battery, forced shutdown, lost/regained UPS comms, replace battery	/etc/nut/upsmon.conf NOTIFYCMD → /usr/local/sbin/ups-notify → ntfy.sh/plex-hgviaey	2026-11-02 18:55 real unplug: ONBATT + ONLINE logged, owner got pushes
Scrutiny	any drive status change	~/torrentvpn/scrutiny/config/scrutiny.yaml → ntfy://ntfy.sh/plex-hgviaey	09-02 push
smartd	health/attribute/self-test failure, temp thresholds (-W)	-M exec /usr/local/bin/smartd-ntfy.sh (bounded, 20 s)	09-03 -M test push
snapraid-sync / snapraid-scrub units	unit failure (precheck abort counts)	OnFailure=snapraid-alert@%n.service → last 25 journal lines to ntfy (ntfy only, no ticket: 10-24..26 failed 3 nights unacted)	09-02 selftest
snapraid-age (parity freshness)	last snapraid-sync run failed, or no sync OK for ≥36 h (dead timer, precheck refusing)	hourly cron-run snapraid-age → job-failure: one page + one Kanboard fix task per incident, closed when green	2026-10-26 all three outcomes run by hand (stub systemctl for the failed branch); cron line not installed yet
rescue-watchdog	archive > 30 h, < 100 MB, stick missing or wrong device	NTFY_URL + WEBHOOK_URL from /etc/rescue-kit.conf (NTFY_URL was empty until 2026-10-11, so it paged nobody)	2026-10-11 test push
rescue-sync	build success	KUMA_PUSH_URL heartbeat + DEADMAN_URL	nightly "heartbeat sent"
precious-sync	copy success	PRECIOUS_KUMA_PUSH_URL heartbeat	nightly
history-backup	any failure, prune > 400 packs, weekly drill FAIL, initial-run completion	ntfy; optional HISTORY_KUMA_PUSH_URL heartbeat	09-08 initial run
store (production, dark)	store-backup failure (dump incomplete, encrypt/R2 failure) or ee-store-prod-health failing a check	erp-backup ntfy + cron-run/job-failure hook; user-timer OnFailure drop-in	2026-10-06 backup + restore drill passed; health check run ok
vpn-check	gluetun egress on the home ISP (Brightspeed/AS19901) or down	Homepage VPN tile red; vpn-status.json	every 5 min
qbit-killswitch	gluetun not running/healthy, or its exit IP equals the home IP (checked every 5th minute), for 2 consecutive minutes	stops qbittorrent; ntfy topic plex-hgviaey on trip and on auto-restart; qbit-killswitch status	every 1 min
tasks-digest	run failed, or did not run at all	OnFailure=tasks-digest-alert → ntfy; a 09:30 canary compares the heartbeat file's date to today	09-13 both directions proven
scheduled user jobs	unit failure for digest, notebook publish, KiCad check, metrics, books snapshot or Kanboard cron	shared tasks-digest-alert.service uses systemd's MONITOR_UNIT to name and quote the actual failed unit	09-20 expansion verified
surfaces-check	an internal page is ungated, blank, has a script that does not parse, or its API is empty	ntfy; twice daily 06:40 and 15:40	09-13 proven by breaking home.js on purpose
postiz-scheduler-watchdog	Postiz's web app is up but its scheduling worker is stalled; failed repair also pages	restart only the in-container orchestrator, verify its health endpoint, then ntfy recovery/failure	09-19 live recovery and controlled container restart
cron jobs (22 of them)	non-zero exit, or the job stops running at all	cron-run <name> pushes ntfy with the last 15 lines + log path and writes a heartbeat; cron-watch.timer reads the heartbeats hourly and pages for one older than its limit	09-23 both directions proven live
off-box watchdog (Cloudflare Worker watchdog.embeddedera.com, deployed 2026-10-09; no failure drill yet)	store/site non-200 or no server heartbeat 20 min	ntfy OFF-BOX: ... from Cloudflare, survives Plex/power/ISP loss; see offbox-watchdog.md	—
sec-watch (SOP-049)	router forwards off the allow-list, new wildcard listener, miner-looking process, pool/attacker connection, Plex exploit traces, old Plex build, sustained container CPU	cron */5 via cron-run sec-watch → ntfy SECURITY <check> (once, repeat after 24 h)	2026-10-18 live push proven with a fake listener; see security-watch.md
uptime-kuma	any monitor down	whatever is configured in Kuma (uptime-kuma container)	—

Silent by design or by omission (know this): - Still unwrapped on purpose: vpn-check and erp-output-clean (every 1–5 min, tile/log-only), precool.py, the SOP-013 feedback line (its && exits non-zero on the normal "nothing to report" path) and engineering-deploy (flock -n exits 1 on ordinary lock contention). Wrapping those would page for healthy behaviour. - rsnapshot — no alerting; check /mnt/disk1/Storage-snapshots/ mtimes. - docker-backup.sh — logs only; rescue-sync would fail downstream if a snapshot were missing. - training-sync — now wrapped as training-evening/training-prealarm/training-morning, so a failure pages; its own usage-limit sleep/retry is unchanged. Formerly: log only. A usage-limit rejection makes it sleep until the reset and retry until the mode's deadline. - hd-idle, wg watchdogs — journal only. - A precheck abort at 03:30 is a failure of the sync unit and therefore pages. Good — but it also means the array is unsynced until someone reads the alert.

2026-09-23 — cron failures and absences now page

cron has no OnFailure, so every script had to remember to alert for itself on every exit path. weekly-research remembered on one branch and not the other, and the 06:00 engineering note silently did not publish; Angel found out by noticing it was missing. Of 19 cron scripts, 10 had no push path at all.

~/.local/bin/cron-run <name> <command> now wraps 22 crontab lines. It tees the job's output (the existing >> …log redirects still fill, unchanged), exits with the job's own code so cron semantics are untouched, pushes ntfy with the last 15 lines on failure, and writes a heartbeat to ~/.local/state/cron-run/<name>.json. Repeat failures are quiet for an hour, or a broken 15-minute job would send 96 pages a day.

~/.local/bin/cron-watch (systemd user timer, *:25) reads those heartbeats and pages for a job that stopped running at all — the case a failure alert can never cover, because a dead job and a quiet day look identical. It discovers jobs from the crontab itself, so wrapping a new job is the only step; only a non-daily cadence needs a line in ~/.config/cron-watch.conf (default 26h). It pages once per job per 12h. Its own failure goes to the shared tasks-digest-alert.service, and SuccessExitStatus=0 1 keeps a mere "late job" finding from also firing that handler.

Names are per schedule, not per script: training-sync runs three times a day and history-backup/erp-backup twice, and a shared name would overwrite the heartbeat. Heartbeats were seeded at install with status: seeded — that records when watching began, not a run.

Backup of the pre-change crontab: ~/.local/state/cron-run/crontab.backup-20260923-133926 (0600). Proven live on 09-23 in both directions: a deliberate exit 4 paged, and an artificially aged heartbeat produced "cron: 1 job(s) not running".

2026-09-23 — a finding is not a crash

Four user units sat in failed and none of them had paged. Two were one-off content-autonomy-*-trial experiments from the build. One was ee-store-order-sync, killed by SIGTERM on 09-22 and deliberately disabled since — the store is not live (Stripe test mode, charges_enabled=false, and the Woo→InvenTree bridge is being rebuilt as launch item 3), so its timer is off on purpose, not broken.

The fourth was real: content-schedule-audit exits 1 to mean "social coverage needs attention", which it also pushes to ntfy itself. systemd read that as a failure, so the unit stayed failed permanently and systemctl --user list-units --state=failed became unreadable — which is exactly how a genuine crash would go unnoticed. It now sets SuccessExitStatus=0 1, the same pattern cron-watch.service uses: the finding still alerts, the unit does not lie. Verified: Result=success, ExecMainStatus=1, is-failed reports inactive.

All four were reset. The board is clean, so "failed" means something again.

Connected to

• joins: schedules, storage/rescue-usb, storage/disks

If you change this

• Hits: the ntfy topic string appears in four places: scrutiny.yaml, smartd-ntfy.sh:12, snapraid-alert@.service, /etc/rescue-kit.conf NTFY_URL — change all four.
• Does not hit: the Kuma heartbeats (separate URLs, in rescue-kit.conf).

2026-09-20 correction (superseded 2026-10-23, see below)

The shared user-unit failure handler formerly labeled every caller as “Task digest failed” and quoted the digest journal. It now reads MONITOR_UNIT, so the title and journal excerpt identify the real failing service. The 05:41 alert was KiCad validation (114 existing design errors), not a digest failure.

See

• Source: /usr/local/bin/smartd-ntfy.sh · /etc/systemd/system/snapraid-alert@.service · /usr/local/bin/rescue-watchdog.sh:46-59 · ~/.local/bin/vpn-check

2026-10-06 — failed jobs: re-run, then one alert, one task

cron-run no longer pages straight away. On a non-zero exit it calls job-failure hook: a transient failure (network/DNS, timeout, HTTP 429/5xx, rate limit, empty-or-not-JSON model reply, busy lock) of a job not on the deny-list is re-run unchanged after ~4 min (max 2 per job per day) and pages nobody if it recovers. Anything else pages once per incident (ntfy with click link + app push, link https://home.embeddedera.com/app#job/<id>) and opens the one Kanboard fix task (same logic as the job page's "Ask Claude to fix"). Detection is the hook, user-unit OnFailure=job-failure@%n.service, and a 10-minute sweep. If the hook cannot handle a job (not catalogued, tool broken, the sweep itself) cron-run falls back to the legacy ntfy + push below, so a broken handler can never silence a failure. Incidents: job-incidents; weekly summary Monday. See ../../processes/job-failure-handling and SOP-043.

2026-10-11 — red on the board must mean someone was told

rescue-watchdog failed 2026-10-08..10 (the rescue stick was full, see ../storage/rescue-usb) and paged nobody, for two reasons. /etc/rescue-kit.conf had NTFY_URL= empty, although this card claimed it alerted; and the job-failure sweep only watched user cron and user timers, so a root timer could sit red in the app indefinitely. NTFY_URL is now set (test push sent). Branch sandbox-system-job-alerts in ~/assistant makes the sweep read root timers too: alert once and open the fix task, never re-run, and skip units that have their own OnFailure= (snapraid, maint). Not live until that branch is released. Rule: when a job's alert path is a config value, check the value, not just the code that reads it.

Social pause (2026-10-16)

While Facebook/Instagram is paused the alerts from story-publish-due, story-handoff, story-preflight, content-approve-missing, postiz-scheduler-watchdog, content-delivery-check and content-schedule-audit are silent because those jobs are off. content-daily-release still alerts for the website stage only. x-post-prompt, social-insights and marketing-snapshot are unchanged.

2026-10-23 — one page per incident for user timers

The shared tasks-digest-alert.service (OnFailure of books-snapshot, cron-watch, kanboard-cron, kicad-check, notebook-publish, tasks-metrics, tasks-digest) was retired. Two faults: systemd skips MONITOR_UNIT when more than one unit lists the same OnFailure handler ("multiple trigger source candidates … skipping"), so 20 alerts since 09-15 were titled unknown-user-job.service with "-- No entries --"; and it paged every failed run, so a 10-minute timer paged every 10 minutes (07:07–08:27 and 14:08–15:18 on 10-23, both notebook-publish refusing to build on a secret-scanner match). All seven are catalogued, so job-failure@%n (global service.d/10-job-failure.conf) now pages them once per incident and opens the fix task. The curl lives on as tasks-digest-alert@.service, wired only as job-failure@'s OnFailure (job-failure@.service.d/20-no-recursion.conf): it pages "job-failure handler broke on " if the handler itself crashes. Backups: ~/.local/state/unit-backup-2026-10-23/. Still open: hook-unit ignores non-catalogue units (claude-remote failed 265× unpaged).