Skip to content

job-fixer — AI fix PRs for escalated job incidents (week 1 of the trust ladder: PR only, owner approves)

Approved by the owner 2026-10-09. Separate from job_failure.py, which stays AI-free. Last seen: cron-run log 2026-10-14 08:30 "no unhandled escalated incident", rc=0. Trust ladder and rules: reference/agent-team-playbook-grok.md, reference/agent-harness-loops-guide.md.

Parts

Piece Where
program ~/.local/bin/job-fixer [--dry-run] (379 lines, ops-scripts); tests ~/.local/bin/test_job_fixer.py (pure state logic) and test_erp_backup.py (the r2_prune keep/delete test, 2026-10-10) — both pass 2026-10-14
trigger crontab */30 * * * * cron-run job-fixer timeout 30m job-fixer
state ~/.local/state/job-fixer/state.json (incident id → attempts), work/<incident> throwaway clones, notes/<incident>.md diagnoses, and notes/<incident>-tryN-<model>.md per-pass reasoning (job-fixer:174)
read-only context every pass also gets --add-dir ~/.local/state and ~/Notes/map (CONTEXT_DIRS, job-fixer:34; ~/.config, .ssh, api-credits and feedback state are denied) so a failure caused by another job can be traced; added after the 2026-10-10 R2 incident
kill switch touch ~/.config/job-fixer.disabled (exit 0)
budget api-credits check fixer before each model call; every call runs as api-credits run fixer -- claude -p

Flow

  1. Pick the oldest open, alerted incident of a catalogued job with < 3 attempts (one per run). At 3 attempts: one ntfy "gave up", then ignored.
  2. In scope only if the job's script (resolved path) is a file in ~/.local/bin (EmbeddedEra/ops-scripts), ~/assistant (EmbeddedEra/assistant) or ~/story-publisher (EmbeddedEra/story-publisher, tests python3 -m unittest discover -s tests -q). Otherwise a no-tools diagnosis note + one ntfy.
  3. Fresh clone from GitHub; maker --model sonnet, 8 turns, tools Read/Edit/Grep/Glob + the repo's tests. Harness runs the tests; no diff or red tests => one escalation to opus with a written handoff (receipts, test tail, diff), not the transcript. Git repos the job's script names by absolute path (/home/angel/<repo>, e.g. ~/story-publisher) are added read-only via --add-dir and the brief says not to edit them; the maker reports file:line there instead (reason: 2026-10-13 story-publish-due, the bug lived in ~/story-publisher). The diagnosis path (job outside an owned repo) gets the same repos with Read/Grep/Glob only, 6 turns; with none it stays no-tools. 3a. Dependency PR pass (2026-10-13): if the own-repo pass ends with no approved PR (no diff, red tests, or checker reject) and a dependency repo the script names is one of the three repos above, ONE more pass runs in a fresh clone of it (work/<incident>-dep) with the same pipeline (fix_pass(): maker, tests, one opus escalation, separate checker, fix/<job>-<date> PR with the same label via --repo). Its brief carries the first pass's receipts as evidence only. Reason: the maker twice diagnosed the bug inside ~/story-publisher but could only say "fix belongs there".
  4. Separate checker call (no tools) must answer APPROVE. Only then: push fix/<job>-<date>, gh pr create with label "job-fixer week-1: owner approves", ntfy "Fix PR ready" clicking to the PR. 4a. Destructive fixes need a test: anything that deletes, prunes, overwrites, sends or spends must ship a small assert-based test (fake input, temp dirs) of what is kept vs removed; the checker REJECTs without it (job-fixer:240, :300). A pass with no diff sends a diagnosis ntfy and stops (no 2nd/3rd spin).
  5. Never: deploy, restart, touch crontab/secrets, work in a live checkout.

2026-10-17 changes (content-buffer-fill incident 1792188424)

All four model passes saved (no output) and the incident was closed as "diagnosis sent". Now: model() returns [model returned no final text: <subtype>, N turns] instead of an empty string; a nodiff run whose only text is that marker is NOT marked done (retried up to MAX_TRIES, then the give-up ntfy); the context block lists the names of pause/disable flag files in ~/.config (the model may not read that folder). Tests in test_job_fixer.py. Cause of the empty replies is inferred (max-turns), not confirmed.

If you change this

  • Hits: api-credits budget jobs.fixer, cron-watch.conf (computed limit used; no entry), jobs.json catalogue (run jobs-index).
  • Not done: Kanboard comment (no standalone helper in jobs_api). The older cron job also called fixer is unrelated (its heartbeat is fixer; this one is job-fixer) but shares the fixer api-credits share.
  • Rollback: kill switch file, then remove the crontab line (backup ~/.local/state/crontab.bak.2026-10-09-fixer holds the prior crontab).

Log

  • 2026-10-17: first three week-1 PRs merged by the owner (story-publisher #2 profile-screen wait, ops-scripts #2 quote-drafts "session limit" regex, ops-scripts #3 onboard GET retry on 5xx); ops-scripts #1 closed as stale. Open PRs now show on Home "Needs you" (see job-failure-handling card). Follow-up: quote-drafts has no overlap lock; a limit sleep of up to 4 h can overlap the */30 runs.