Fixes

Incident 679: pod-pin-feeds — escalated

The 2026-11-11 09:45 run built the feeds, then the `npx --yes wrangler@4 pages deploy` call in `_deploy_feeds` (pod/run.py:1106) exited 1. The 18:45 run on 11-10 deployed the same way and succeeded, so this is not a code regression. The log tail shows no wrangler stderr, so the actual wrangler error is unknown. Likely causes are an expired or revoked Cloudflare API token, a Cloudflare or npm outage, or a wrangler@4 version change picked up by `npx --yes`. `subprocess.run` with `check=True` raises without capturing wrangler's output, which is why the log lacks the reason.
[]

Incident 674: coach-proposal-163 — proposal

Not verified against code: I had no file or shell access in this run, so this is inferred from the evidence alone. The listing audit's vision check failed a ranked mug image because the listing's image set shows only a single front-facing mockup. The wrap-around artwork is wider than the visible face of the mug, so its left and right portions are out of view, and no other image in the set (rotated views or a flat artwork preview) shows them. The likely root cause is that the mug mockup step produces one camera angle regardless of artwork width.
[{"at": 1794402321.3800428, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 673: coach-proposal-162 — proposal

Not verified against the code: this session had no file or shell tools, so the diagnosis rests on the evidence text alone. The listing audit's vision check flagged a mug photo where the wraparound artwork continues past the visible right edge. The most likely cause is that the listing's image set is rendered from a single camera angle (or too few), so wide wrap designs are never shown in full. The vision check itself appears to be working correctly; the gap is in which mockup views get generated.
[{"at": 1794402026.479179, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 672: coach-proposal-161 — proposal

Unverified hypothesis, no repo inspection was possible in this run (only the evidence payload was available). The listing audit's vision check flagged a mug listing photo: the mug itself is fully in frame, but the boots-and-shirt artwork wraps around the mug, so a single front-facing mockup cuts the design off at the left and right edges. The likely cause is that the listing image set for mugs has only one camera angle and no flat artwork image, so no photo shows the complete design. This looks like a gap in the mockup/photo generation step of the listing publisher, not a fault in the audit.
[{"at": 1794401722.3382802, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 659: coach-proposal-193 — proposal

The audit and snapshot checks both flag tote listings with no sibling link to the matching sticker listing. The likely cause is missing sibling/related-product data on those tote records, or a link generator that doesn't pair tote and sticker. I had no file or shell access in this session, so I did not inspect any code or data. I can't confirm which repo owns the listings or which of those two causes applies.
[{"at": 1794373218.0554466, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 642: coach-proposal-192 — proposal

Not determined. The evidence only shows the symptom: the listing audit and the snapshot check both flag poster listings with no sibling link to the matching sticker (reported five times). I had no file or shell access in this run, so I did not inspect either repo. The repeat count across two independent checks suggests a systematic gap in the publishing step that writes sibling links (poster -> sticker), rather than a one-off data slip, but that is an inference from the evidence, not a verified finding. Possible causes, all unconfirmed: (a) the publisher never emits the sticker sibling link for posters, (b) the sticker listing does not exist or is not yet published when the poster is generated, so the lookup comes back empty, (c) the poster/sticker matching key (slug or story id) does not line up.
[{"at": 1794315322.0040152, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 641: coach-proposal-160 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1794315016.526114, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1794401414.8858294, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 637: coach-proposal-191 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1794286830.209972, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1794373521.2368424, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 635: coach-proposal-190 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1794229523.4388988, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1794315922.856601, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 634: coach-proposal-189 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1794229220.3430707, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1794315617.974198, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 630: coach-proposal-188 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1794200713.458213, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1794287122.304123, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 627: pod-orders — resolved

The last log entry is a "start" at 2026-11-08T08:00:01-05:00 with no matching "end rc=" line. Every earlier run finished in 2-6s with rc=0 and "new_orders": []. So the 08:00 run either hung or was killed before it could write its end line, or the log was read while it was still running. The job is flagged late, which fits a stalled run. The evidence has no error text, so the cause is unconfirmed. Likely candidates are a hung network call in `pod.run orders` with no timeout, or the process being killed (reboot, OOM, or the timer's stop).
[]

Incident 626: coach-proposal-187 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1794142513.180787, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1794228922.0003023, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 625: coach-proposal-63 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1794142232.111872, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1794228629.6143665, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 623: coach-proposal-185 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1794114037.277062, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1794200429.3231184, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 621: coach-proposal-184 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1794056729.6781077, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1794143413.085323, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 620: coach-proposal-183 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1794056113.2128117, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1794143134.7685554, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 617: coach-proposal-186 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793970037.9910934, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1794114329.4647899, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 616: coach-proposal-182 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793969734.013154, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1794056418.5422935, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 615: coach-proposal-167 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793969433.4216268, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1794055832.2981305, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 611: coach-proposal-200 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793941235.9987836, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1794027930.4551265, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 609: coach-proposal-199 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793883933.062226, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1794027638.9717798, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 608: coach-proposal-195 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793883031.6665947, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793970317.6457012, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 606: coach-proposal-202 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793854842.908391, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793941527.652629, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 602: coach-proposal-198 — escalated

Proposal stopped: ValueError: change must contain 1–24 files
[{"at": 1793797239.0925224, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: forbidden command or production mutation in diff"}, {"at": 1793883644.6350398, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: change must contain 1\u201324 files"}]

Incident 601: coach-proposal-197 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793796637.5108914, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793883326.8345737, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 593: coach-proposal-196 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793768730.0730186, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793855131.7886589, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 575: pod-laser-orders — resolved

Repair stopped: ValueError: runbook retry refused: job is not retry_safe
[]

Incident 563: pod-laser-orders — resolved

Repair stopped: ValueError: runbook retry refused: job is not retry_safe
[]

Incident 551: pod-laser-orders — resolved

Repair stopped: ValueError: runbook retry refused: job is not retry_safe
[]

Incident 539: pod-laser-orders — resolved

Repair stopped: ValueError: runbook retry refused: job is not retry_safe
[]

Incident 527: pod-laser-orders — resolved

Repair stopped: ValueError: runbook retry refused: job is not retry_safe
[]

Incident 516: pod-laser-orders — resolved

Repair stopped: ValueError: runbook retry refused: job is not retry_safe
[]

Incident 505: pod-laser-orders — resolved

Repair stopped: ValueError: runbook retry refused: job is not retry_safe
[]

Incident 488: coach-proposal-137 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793710839.2439868, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793797534.1735632, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 486: coach-proposal-136 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793710234.4087691, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793796940.065718, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 482: pod-pin-feeds — resolved

The 2026-11-02 18:45 run of `pod.run pin_feeds` logged a start line but never an "end rc=" line. Every earlier run finished in about 80s with rc=0. At the evidence time (about 09:46 on 11-03) that run had been open for roughly 15 hours, and the 09:45 run on 11-03 had not logged either. So the process either hung or was killed without the wrapper writing an end line, for example by a reboot or an OOM kill. The most likely hang is a network call with no timeout, either to the oddlyfittinggoods.com feed fetch or to the Pinterest API. A stuck instance may still be running and blocking later runs through a lock or overlap guard. I did not inspect the process list or the code, so this is inferred from the log alone.
[]

Incident 455: coach-proposal-135 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793625049.6201818, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793768437.8741896, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 454: coach-proposal-134 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793624715.70435, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793711427.279941, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 453: coach-proposal-133 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793624446.4426093, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793711131.8803198, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 443: coach-proposal-159 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793595929.150094, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793682627.0506153, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 442: coach-proposal-90 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793595640.1491725, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793682326.9766436, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 440: coach-proposal-61 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793537433.9313655, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793623828.5083525, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 439: pod-orders — resolved

The log shows no failure. Every run through 2026-10-31 18:00 EDT ended rc=0 in 2-5s with "new_orders": []. The epoch 1793484006 matches the last run's end (18:00:06 EDT, 22:00:06 UTC). "late: true" therefore only means the next 08:00 run on 2026-11-01 had not been logged when the evidence was gathered. Two explanations fit this evidence: (a) the check ran before the 08:00 run, or the lateness threshold is too tight; (b) the DST fall-back at 02:00 on 2026-11-01 shifted a local-time cron or timer, or the lateness calculation. I can't tell which from the evidence. Nothing here shows a code bug in pod.run orders.
[]

Incident 437: pod-scout — resolved

No failure is shown. The last completed run was 2026-10-31 21:00 (rc=0, epoch 1793494823 matches its end time). The log then ends with a "start" line for 2026-11-01T06:00:01-05:00 and no "end" line, so the job is flagged late because that run was still in progress when the evidence was gathered. The 06:00 run is the slow one: it does the fast_shops pass and took 586-646s on 10/28-10/31, versus about 20s for the other runs. The wall clock also moved from -04:00 to -05:00 at the DST change, which may have skewed a lateness check that compares times. I can't tell from this log alone whether the run later finished or hung.
[]

Incident 432: coach-proposal-132 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793448038.3212788, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793535035.830938, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 431: coach-proposal-131 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793447726.2835298, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793534127.429314, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 430: coach-proposal-114 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793447414.9256134, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793533836.6263304, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 429: coach-proposal-156 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793419530.1557038, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793506232.5511422, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 428: coach-proposal-155 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793419250.7798502, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793505933.7899802, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 426: coach-proposal-154 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793361036.639933, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793448330.7579172, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 424: coach-proposal-153 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793275534.1639893, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793361917.1945045, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 423: coach-proposal-151 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793275248.7438824, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793361640.6247137, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 422: coach-proposal-150 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793274634.8633783, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793361338.416038, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 421: coach-proposal-149 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793246732.4891343, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793333431.561184, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 419: coach-proposal-148 — escalated

Proposal stopped: ValueError: systemd-run failed (exit code 226)
[{"at": 1793189412.9974875, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793332835.4591007, "action": "code_fix", "detail": "systemd-run failed (exit code 226)\nBranch: fixer/coach-proposal-148-20261030-419-2"}, {"at": 1793332835.4734316, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: systemd-run failed (exit code 226)"}]

Incident 418: pod-orders — resolved

The 2026-10-28 08:00:02 pod-orders run logged a start line but never an "end rc=" line, and it printed no JSON. The previous 40 or so runs each finished in 2-4s with rc=0. The job is flagged late because the last completed run was 2026-10-27 18:00. So the process either hung or was killed mid-run. The usual suspect is a blocking network call to the print-on-demand provider API with no timeout. The wrapper's end line is also not written when the process is killed. The evidence has no traceback, so the exact cause is unproven.
[]

Incident 417: coach-proposal-147 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793188237.569207, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793274938.6588812, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 416: coach-proposal-145 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793160028.2319956, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793246420.7725816, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 411: coach-proposal-144 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793102427.8508801, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793189112.3835533, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 410: coach-proposal-143 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793102112.3856804, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793188534.465826, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 400: coach-proposal-142 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793073925.8185956, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793160623.165047, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 392: coach-proposal-141 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793016330.7097707, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793102731.6818361, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 391: coach-proposal-140 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1793015426.535698, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793101813.4624717, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 374: coach-proposal-139 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1792929921.062016, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793073634.251157, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 373: coach-proposal-123 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1792929620.5103292, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793016020.425177, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 372: coach-proposal-122 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1792929313.2807853, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1793015734.5105143, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 367: coach-proposal-31 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1792842613.7682495, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792929012.9003356, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 356: coach-proposal-121 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1792757714.760711, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792844415.0967607, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 355: coach-proposal-120 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1792757413.3152041, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792843832.6571815, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 354: coach-proposal-119 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1792756230.0372071, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792842913.9371579, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 352: coach-proposal-117 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1792670415.0503378, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792757125.7326567, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 351: coach-proposal-116 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1792670112.5192976, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792756816.2942343, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 350: coach-proposal-115 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1792669814.787278, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792756512.6104472, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 339: coach-proposal-152 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1792584637.9362988, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792671310.714635, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 338: coach-proposal-146 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1792584035.1857114, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792671027.2900474, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 337: coach-proposal-138 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1792583438.6546235, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792670725.051383, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 335: coach-proposal-113 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1792498530.2338061, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792584980.6943104, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 334: coach-proposal-112 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1792497929.3873553, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792584317.807398, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 333: coach-proposal-111 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1792497328.4385474, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792583714.820917, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 329: coach-proposal-110 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1792411558.504444, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792498227.978003, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 328: pod-orders — resolved

The log shows no failure. Every pod.run orders run since 2026-10-05 ended rc=0 in 1-4s with new_orders empty. The newest entry is the 2026-10-19 08:00 run, whose end time matches the evidence epoch (12:00:03 UTC). The "late" flag therefore means the expected 18:00 run on 2026-10-19 has not logged a start or end yet. Either the check ran before that run fired (the checker's grace window is too tight), or the scheduler skipped the run. The evidence has no scheduler state, so I can't tell which. It points to no code bug in pod.run orders.
[]

Incident 327: coach-proposal-109 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1792410925.901071, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792497633.6233902, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 326: coach-proposal-108 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1792410624.9971857, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792497037.251783, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 324: coach-proposal-130 — escalated

Proposal stopped: ValueError: source repository has uncommitted changes
[{"at": 1792346741.89086, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792433433.733762, "action": "code_fix", "detail": "source repository has uncommitted changes\nBranch: fixer/coach-proposal-130-20261019-324-2"}, {"at": 1792433433.73755, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: source repository has uncommitted changes"}]

Incident 323: coach-proposal-107 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1792346434.9570143, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792433128.4245477, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 322: coach-proposal-106 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1792346130.381343, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792432849.6472292, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 314: pod-coach — resolved

The job failed again after the repair
[{"at": 1792324202.9632742, "action": "wait_and_retry", "detail": "Retry scheduled once in 15 minutes"}, {"at": 1792325406.0888186, "action": "retry", "detail": "hq-retry-pod-coach-1792325406"}]

Incident 308: pod-coach — resolved

The job failed again after the repair
[{"at": 1792237801.9385717, "action": "wait_and_retry", "detail": "Retry scheduled once in 15 minutes"}, {"at": 1792239003.9521165, "action": "retry", "detail": "hq-retry-pod-coach-1792239003"}]

Incident 260: coach-proposal-105 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1792152018.3428302, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792240214.0432546, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 259: coach-proposal-104 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1792151726.2238975, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792239911.7163663, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 258: coach-proposal-101 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1792151430.3115628, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792239626.2358296, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 254: coach-proposal-100 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1792123233.1460886, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792209927.7108748, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 246: coach-proposal-99 — escalated

Proposal stopped: ValueError: Diagnosis was missing or invalid
[{"at": 1792065912.1624398, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792152613.24765, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Diagnosis was missing or invalid"}]

Incident 245: coach-proposal-98 — escalated

Proposal stopped: ValueError: Diagnosis was missing or invalid
[{"at": 1792065633.353309, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792152311.3649242, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Diagnosis was missing or invalid"}]

Incident 230: coach-proposal-97 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1791979513.9297194, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792066225.0095606, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 229: coach-proposal-96 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1791978924.1171198, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792065324.8251722, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 228: coach-proposal-95 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1791978626.5173826, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792065013.608344, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 208: coach-proposal-118 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1791893728.6790452, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1792036820.0757267, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]

Incident 207: coach-proposal-103 — escalated

Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis
[{"at": 1791893123.2093482, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}, {"at": 1791979828.0408287, "action": "proposal_stop", "detail": "Proposal stopped: ValueError: Proposal needs a medium/high confidence code diagnosis"}]