job-failure-handling — what a failed scheduled job triggers

Detection, unchanged re-run, notification, ticketing and logging. Never invokes Claude/Codex/any agent, never changes code, never pushes (owner constraint 2026-10-06). Procedure: SOP-043.

Parts

Piece	Where
logic (classify, policy, fold, sweep, retry, report)	~/assistant/job_failure.py (tests tests/test_job_failure.py); fix task = jobs_api.ensure_fix_task
entry points	~/.local/bin/job-failure hook|hook-unit|sweep|retry|report; ~/.local/bin/job-incidents (open incidents; --all, --json, fix <job> <commit>)
trigger 1	cron-run calls job-failure hook on non-zero exit (exit 0 = handled, legacy alert skipped; else legacy alert)
trigger 2	~/.config/systemd/user/service.d/10-job-failure.conf OnFailure=job-failure@%n.service for every user service; hook ignores non-catalogue units
trigger 3	crontab */10 * * * * cron-run job-failure-sweep (safety net, escalates deferred, closes green incidents)
weekly	crontab Mon 07:20 cron-run job-failure-report → app notify + ntfy
log	incidents.jsonl (append-only events; incidents are folded, no other state)

Decisions

• Transient = DNS/network, timeout, HTTP 429/5xx, rate limit, "empty or not JSON", lock/database busy. Hard errors win over transient words. Unknown = permanent.
• Re-run after 240 s via systemd-run --user --on-active, max 2 per job per local day, through the job's own crontab command (or systemctl --user start), with JOB_FAILURE_RETRY=1 so cron-run stays quiet and the re-run reports its own outcome.
• Frequent jobs (cron every ≤15 min) are deferred 30 min instead of re-run.
• Deny-list = DENY in job_failure.py (see SOP-043); default for an unlisted job is re-run.
• The fixer cron job (separate, pre-existing) still reads heartbeats and may retry/diagnose on its own; this process does not use it.

If you change this

• Hits: cron-run (hook block), jobs.json catalogue (hook only knows catalogued jobs; run jobs-index after adding one), cron-watch limits for job-failure-sweep/-report.
• Rollback: delete the hook block in cron-run (legacy alert returns), remove the crontab lines and service.d/10-job-failure.conf.

Fixer PRs on Home (2026-10-17)

fixer-prs-refresh (cron */15, ops-scripts) lists open PRs labelled "job-fixer week-1: owner approves" in ops-scripts, assistant and story-publisher into ~/.local/state/cockpit/prs.json. Home's "Needs you" card (admin only) shows them as "Fix PR waiting: ", folded into one row when the card is full. A file older than 30 min shows "Could not check fix PRs" instead of stale rows. Status: live since 2026-10-17 10:01 (assistant main 485df98; the same restart shipped the social-pause edits).

Frequent jobs defer on any failure (2026-10-25)

A job that reruns within 15 min (is_frequent) now defers on any class, not only transient: the next run is the retry and the sweep escalates after 30 min if it stays red. Deny-listed frequent jobs keep the old rule (permanent → escalate). Cause: booking-warm paged on one silent curl exit 23. assistant cf870a8.

Weekly report (2026-10-25) appends storage_line(): reads ~/.local/state/storage-forecast/summary.json + reclaim.jsonl (assistant 4a9cbba).