Skip to content

Off-box watchdog (ee-watchdog) — deployed 2026-10-09

A Cloudflare Worker with a 10-minute Cron Trigger that checks the store, the marketing site and a server heartbeat from outside the house, so a dead server, power or ISP still pages. Code written 2026-10-08, deployed 2026-10-09 (commit 3eb6f0c, KV id set in wrangler.toml); live at watchdog.embeddedera.com (unauthenticated / and GET /beat answer 404/401 — worker.js:61-66).

Why this shape

Every other alert runs on Plex, so none survive Plex dying. The store is tunnel-served (CNAME to cfargotunnel), so a 200 plus marker proves tunnel and origin without touching Access. A push heartbeat (cron-run offbox-heartbeat, 5 min) covers a wedged host; no Access bypass is involved.

Shape

  • Checks: store 200 + marker EmbeddedEra Store, embeddedera.com 200, heartbeat < 20 min old.
  • Alert after 2 consecutive failures, once per outage, plus recovery; ntfy plex-hgviaey, Title OFF-BOX: <what> down.
  • State in KV (state, beat); secret BEAT_SECRET as a wrangler secret.
  • Server side, live: crontab */5 * * * * cron-run offbox-heartbeat offbox-beat (~/.local/bin/offbox-beat reads the secret and pipes the header to curl on stdin) (crontab -l, ~/.local/bin/crontab.txt:107) with the secret in ~/.config/cloudflare/offbox-beat-secret (0600), and offbox-heartbeat = 1 in ~/.config/cron-watch.conf:67. Last run ok, ~/.local/state/cron-run/offbox-heartbeat.json (2026-10-14 08:35). Deploy steps: repo README.
  • Not proven: an alert has not been seen firing (no failure drill logged). Fixed 2026-10-15: cron-run logs the full command line, which used to carry the expanded secret (845 copies in the heartbeat log). The secret now never touches a command line; BEAT_SECRET was rotated (64 hex chars, no trailing newline — a newline uploaded with wrangler secret put breaks the compare) and the old log was emptied. Rule: never pass a secret as a cron-run argument.

Citations: /home/angel/embeddedera-watchdog/README.md

Connected to

  • owns: ~/embeddedera-watchdog/
  • owned-by: alerting.md
  • joins: cron-run, cron-watch, Cloudflare tunnel

If you change this

Re-run node step.test.mjs; keep the 2-failure rule; never add an Access bypass for the heartbeat.