Skip to content

Tester → ERP backlog: boards tested but not finished in the build orders (2026-10-21)

Found by a read-only check from Angel's Mac (nothing on the server was changed). For the session that works on the tester API and InvenTree on Plex. Production data: verify each step before writing, never touch the sandbox ERP's down, and do not double-consume material.

Symptom

Angel tested about 25 Robot Cow boards to FINAL PASS and the build order shows only one finished. Tester DB (/home/angel/.local/state/tester-api/tester.sqlite3, read-only): - 23 distinct boards have a post_coating PASS since 2026-10-14; only 2 of them are mirrored (erp_status = done). - Part 181 builds: BO-0001 (pk 2) Complete 27/27; BO-0002 (pk 3) Production 9, completed 0; BO-0003 (pk 5) Production 9, completed 0; BO-0005 (pk 7) Production 9, completed 1; BO-0004 (pk 6) Cancelled. - All events are accepted, the station outbox is empty and nothing was rejected, so the station side is fine. - Pending events today: 112 (test_run 75, serial_assigned 36, quarantine 1).

Two separate blockers

1. Build orders are full (36 serial_assigned and what hangs off them). erp_error = "all 3 open build orders for part 181 are full; create or issue another". BO-0002, BO-0003 and BO-0005 each already hold 9 serialized outputs (serials.erp_build_pk 3/5/7: 9 each, plus 27 on pk 2), so a new serial has no build to attach to; 52 serials were assigned today and only 16 got a stock item. The dependent events then wait with "no ERP unit yet for this chip/serial (waiting on serial assignment and a pre-coating PASS)" (33 of the 43 pending post-coating PASS events).

2. Two material-consumption operations stuck at started, which block every other unit on builds 3 and 5. erp_operations: material:3:374:stage:pretest and material:5:408:stage:pretest are started, and pretest-required:374 / :408 are required. Stock item 374 is serial 10260007 on build 3 (quarantined 2026-10-14 20:23), stock item 408 is serial 10260016 on build 5 (quarantined 2026-10-21 08:39); both show status Quarantined. The quarantine events carry "material consumption pending: exact allocation state not confirmed; no retry submitted". While an operation on a build is unconfirmed the API refuses to start another consumption on that build: "another material consumption for this build is pending verification" (15 events, including 10 post-coating PASS events). This is why BO-0002 and BO-0003 finished nothing, while BO-0005 (no stuck operation) finished one.

What to do (in this order)

  1. Resolve the two stuck operations. For stock items 374 and 408, check in InvenTree whether the pretest materials were actually consumed for that output (build item allocations and consumed stock, build completed / output history). If consumed, mark the operation done with the evidence; if not, run the consumption once. Do not resubmit blindly: the guard exists to prevent double consumption. Record what you found in erp_material_evidence.
  2. Add build capacity for part 181. Create and issue another build order (or raise a quantity) with room for the 36 waiting serials plus a margin, matching the existing orders' settings (see BO-0003 / BO-0005). Confirm the matcher (_find_single_open_build or its replacement) picks it up.
  3. Let the pending events drain (find out whether tester-api retries by itself or needs a nudge, and say which). Then check: every board with a post_coating PASS has its output in Finished goods, each BO's completed rises accordingly, events.erp_status has no leftovers for those boards, and no material was consumed twice (compare consumed counts to outputs).
  4. Make it not silent next time (follow-up, ask Angel first): alert when the open build orders for a part have fewer than N free outputs, and when an erp_operations row stays started for more than a few minutes.

Expected end state

About 25 Robot Cow finished in Finished goods, BO-0002/0003/0005 progressing (or complete), quarantined boards 10260007 and 10260016 still in Quarantine with a confirmed consumption record, and the pending count close to zero. Report the numbers to Angel and append a Log line in projects/Universal_Tester.md.