universal-tester¶

Reusable "brain box" + swappable fixture boards for automated PCB test. Repo: ~/Documents/GitHub/Universal_Tester. Board layout work for the next revision, universal-tester-003, was scrapped 2026-10-14; UT002 stays the live tester. Productization, pricing, and the build plan live in universal-tester-product.

Host PC --USB--> UT002 brain box --40-pin 2x20--> Fixture board --> UUT

Three layers, three owners. Brain box owns rails (3V3/5V/0-30V var), measurement (ADS1120 precision path), MCU buses, USB hub, fixture-ID EEPROM read. Fixture owns ID EEPROM, UUT pin mapping, level shifting, golden peripherals, and the test sequence. Host owns discovery, orchestration, bench instruments, serial-number policy, append-only logs, labels, and failure receipts. No layer duplicates another's contract.

Operating it¶

Production run: double-click the launcher, never a hand-picked serial port. Each fixture ships four with the same display name — .command (macOS), .bat (Windows), .desktop (Linux), .sh (shared POSIX runner). First launch builds a private .cow-test-venv; later launches reuse it. Startup validates all four exist via ProductionLauncherSet.validate() before discovery, so a fixture can't silently lose an OS.

Current stations: Cow Robot Test Station, Cow Remote Test Station, plus Cow Remote Diagnostic and Cow Remote Button Diagnostic.

Universal Tester Station (PR #11, branch universal-station, 2026-09-26): one double-click launcher for every fixture — Universal Tester Station.command/.bat/.desktop/.sh in host-api/. It reads the 32-byte fixture record at EEPROM 0x51 (UUT power untouched), checks the CRC and runs the owning station program unchanged: magic CRFX → Cow Robot Test Station, CMFX → Cow Remote Test Station; blank EEPROM asks which fixture; unknown/corrupt refused; then waits for the next fixture. --detect-only identifies without launching. New fixture = one row in universal_tester/production/fixture_station.py PROGRAMS. This is a router over the accepted programs, not the Phase 3/4 shared orchestrator (that remains simulation-only until the Robot bench gate in manufacturing-operating-system-plan).

Prereqs: Python 3.10+, libusb, dfu-util on PATH. macOS: brew install libusb dfu-util (launcher checks both /opt/homebrew/lib and /usr/local/lib). Linux: site-approved udev rules, never run the station as root.

Auto-discovery runs at the start of every attempt and names every missing device instead of dying on a VISA timeout: - Siglent SPD3303X-E — USB 62700:5168 - Siglent SDM3055 — USB 62700:4609 - UniversalTester-002 — USB 0483:5740 + product string - dfu-util on PATH

The DUT is discovered after the fixture enables USB1 — it is intentionally absent at startup. --resource, --dmm-resource, --tester-port exist for diagnostics only.

Control panel (bench/bring-up, not production): double-click Universal Tester Control Panel.command → localhost UI on http://127.0.0.1:8765/. Default ceilings 5.0 V / 0.5 A on SPD CH1+CH2; raise them only by running the backend directly with --max-voltage/--max-current. CH3 is a manual fixed output the panel does not touch. Every button fires immediately, no confirm dialog. A 3 s watchdog reopens dropped USB/VISA sessions and deliberately brings USB, DMM insertion, CH1 and CH2 back off — it never silently re-powers. Raw SCPI can desync the UI; hit ALL OUTPUTS OFF after.

Dev usage:

cd host-api && python3 -m venv .venv && . .venv/bin/activate
pip install -e . && python examples/basic_control.py

Where things live¶

host-api/universal_tester/ — client.py (discovery + thread-safe transport), pins.py (restricted-pin checks), instruments/, station.py, production/ (profile, terminal, launchers, logging, dfu, rework), printers/ (Brother label, Zebra bag label, Citizen CT-S801 receipt), optical_* (camera calibration, ROI, color, registration).
host-api/examples/cow-board-test.py — the reference production program.
host-api/examples/fixture_profile_template.py — start every new fixture here.
universal-tester-002-firmware/ — STM32G0B1, HAL + ST USB device lib, CDC.
universal-tester-002/, fixture-board-cow-*/, power-analog-board/, digital-control-board/ — KiCad.
portable-universal-tester/ — separate ESP32-S3/DM40C/DP100 battery prototype. Different product, different contract; don't leak UT002 assumptions into it.
Logs: host-api/examples/log-cow-remote/, log-cow-robot/ — all-board-tests.jsonl (append-only, source of truth), equipment-usage-summary.json (rebuilt atomically from it, never hand-edited), serial-number-registry.jsonl, printed-label registries, rework actions.

Rules that bite¶

Three versions, never conflated: COW_TESTER_VERSION in examples/cow-board-test.py (currently 2.5.0), firmware_version.h (1.1.0, 2026-08-01), and each fixture's ProductionFixtureProfile. SemVer, never decremented or reused, and when unsure pick the larger bump. See semver-discipline.
Every new fixture defines and validates a ProductionFixtureProfile before discovery or power-on. Shared behavior goes in universal_tester/production; only limits, identity, firmware rules, and board tests go in the fixture program. Do not fork the Cow Robot code.
Every result records software + firmware + fixture + UUT versions.
Never infer a pin from a similarly-named fixture. Section 4 of Universal_Tester_Requirements.md is the locked pinout; cross-check against the current schematic.
Keep logs and the serial registry outside any replaceable app bundle (PyInstaller deployment is the planned next step).

Safety¶

Outputs default off on reset, disconnect, invalid config, and watchdog failure. No UUT power until fixture identity, wiring, limits, and instruments are all verified. Hardware interlocks outrank software intent, always. Use the documented discharge/safe-state sequence before changing routing. Don't hang arbitrary grounded equipment off an energized setup without walking the ground path — the portable tester in particular has no galvanic isolation and requires the Mac on internal battery, charger and unreviewed peripherals disconnected.

Fact precedence when docs disagree¶

safety + explicit user instruction → AGENTS.md → locked requirements/ADRs → current source and tests → examples, logs, backups, history. Historical files are evidence, not authority. A design file existing does not mean the circuit passed measured acceptance test.

Why did the Cow Remote station call /reserve 34 times? (2026-09-19) Bursts of three a second apart, at 8 separate times, burning 656 serials (09260038–09260695, since released). The Cow Robot reserved exactly once from the same Mac and behaved correctly, and the pool persists correctly on Linux, so it looks macOS- or Remote-path specific (three SerialClients per run, or a pool file that isn't being re-read). The server now hands back a station's held serials instead of issuing new ones, so it can't waste the sequence again, but the client still shouldn't do it. Evidence: tester-api SQLite serials.reserved_at, and the service log.
Diagnostics must not consume serials. Some Remote diagnostic run assigned serials to fake chip IDs (USB-A, USB-B, USB-BOARD-A/B) — 4 serials, voided 2026-09-19. Find the caller and have it use a lookup or an explicit dry-run, never assign.

Log¶

2026-09-26: Tester API periodic ERP retry is live (assistant PR #3, merge 32b0e11, restarted 15:47). A lifespan thread inside the API process re-runs _mirror_background every TESTER_ERP_RETRY_SECONDS (default 120; 0 disables), serialized by _MIRROR_LOCK. It is deliberately not a separate systemd timer: that lock is per-process, so a second process could double-post a consume. Verified: journal shows "periodic ERP retry every 120 s"; tester.sqlite3 was written at 15:49:13 and 15:51:21 with no station events; 122 tests pass.
2026-09-26: Station session: Robot pre-coating PASS for 09260042, 046, 059, 060, 061, 062; post-coating PASS for 042 and 046. 046 sat In test because tester_inventory._consume checked before InvenTree's async consume task finished and then waited, since pending events only retry when the next station event arrives. A manual process_pending_events confirmed the exact quantities and completed it; BO-0001 is now 23/27. 09260063 failed pre-coating with no readable chip, then ABORTED; it has no ERP unit or quarantine case.
2026-09-26: Cow Remote 3.14.0 (Universal_Tester PR #11, e97c59f): FAIL offers only Retest/Quarantine; the Citizen receipt prints only on Quarantine; decisions append to remote-dispositions.jsonl. Remote quarantine is still station-only (no server case or Zebra bag label) until Remote becomes a plugin. Shipping firmware provenance: Remote_6.2.4.bin existed only in The_Real_Cow_Flasher (6ea1fe3); Cow_Remote PR #4 adds the byte-identical binary at firmware/releases/, contract-format manifests for 6.2.4 (production) and 6.14.0 (test), and a CI hash check. Its build source commit was never recorded.
2026-09-26: PR #11 (6c59868): Universal Tester Station now refuses a blank fixture EEPROM instead of asking and letting the program engrave it, matching the agreed rule that production never provisions fixtures. Memory universal-tester-pickup rewritten from the full Codex transcript (phases 1-4 status, six agreed station rules).
2026-09-26: Built the Universal Tester Station launcher (PR #11, aa6a71e on universal-station, pushed, not merged — merge without review was blocked for the agent). Both fixture programs share the record layout <4s magic><u8 schema><3×u8 profile><16s id><4s reserved><u32 CRC32> at 0x51, which is what makes routing by magic possible. 5 new tests; full suite 511 + 70 subtests pass. Not yet run on hardware. Checked the same day: all 13 ~/engineering/repos clean and in sync with GitHub; ~/Universal_Tester-quarantine is a leftover worktree of merged sandbox-quarantine.
2026-09-21: Universal Tester PR #10, merged as 33a1bea, moved operator-run Phase 4 rehearsal reports to ~/Documents/EmbeddedEra/UniversalTester/Rehearsals/ with unique UTC filenames so rehearsals never dirty the source checkout.
2026-09-21: Phase 4 software candidate.2 completed. Cow Robot Board PR #14 merged as 3072045, Universal Tester PR #9 as c04b38c, and tester API PR #2 as 9cd4326. Workflow 1.1.0 uses manufacturing-test firmware 4.0.0 through pre/post testing, then production firmware 4.5.0 plus a required smoke test before the final label. Repair routing returns to the original stage unless earlier evidence is invalidated. Eleven no-hardware/no-ERP rehearsal scenarios passed; 49 focused tester tests, 119 API tests, product validation, wheel packaging, cross-repository hash equality, and CI passed. Production API configuration and station launchers remain inactive; physical fixture serial and complete bench acceptance remain.
2026-09-21: Universal Tester PR #8, merged as d5d3e6f, simplified the shared operator display after laptop review: setup details appear once; each test shows only its name, instruction, progress bar, and result; final disposition is uncluttered; color is limited to PASS, FAIL/BLOCKED, and ACTION REQUIRED words. Focused shared/Robot validation passed 27 tests and the rehearsal passed.
2026-09-21: Universal Tester PR #7, merged as 2e90f17, added the shared Host API 1.5.0 operator display: named test steps, progress bar and percentage, green PASS, red FAIL/BLOCK, yellow running, and cyan operator-action states. The Robot hardware-free rehearsal now displays every approved stage step. Focused shared/Robot validation passed 27 tests and the rehearsal passed; production launchers remain unchanged pending bench activation.
2026-09-21: Cow Robot hardware organization completed in Cow_Robot_Board PR #13, merged as aedd64c. The ordered and built 004B design is the sole PCB on main at hardware/Robot-Cow-004B; retired and unreleased designs are preserved on archive/pre-production-hardware-cleanup-2026-09-21. The three commented manufacturer-proven DRC exclusions are preserved and the dedicated KiCad CI passes. Universal Tester PR #6, merged as 5fa3e14, syncs the resolved design-review gate. Physical fixture identity and bench acceptance remain required.
2026-09-21: Phase 4 Robot reference candidate merged: Universal Tester PR #4 at a3cf096, Cow Robot Board PR #11 at 88de848, and tester API PR #1 at 32f9cbd. Added the pinned Robot Cow 004B workflow/fixture/firmware/ERP release package, exact server authority checks, next-stage endpoint, unlimited retest/quarantine session, and physical bench matrix. Automated validation: Universal Tester 501 passed + 1 skip + 70 subtests, tester API 118 passed, manufacturing validators clean, firmware CI passed, and cross-repository sandbox passed. Follow-up PRs Cow Robot #12 and Universal Tester #5 record seven legacy shipping-board ERC/DRC errors as an activation blocker and fix report retention. Candidate is not activated; fixture-serial verification, design-error review, and the complete physical bench matrix remain.
2026-09-21: Manufacturing operating system Phase 3 completed in Universal Tester PR #3, merged as 3f6fdf8. Added the opt-in shared orchestrator, runner, operator UI contract, unpowered fixture and release compatibility gate, append-only attempt journal, end-to-end event-ID preservation, tester-server adapter, legacy program adapter, and hardware-free simulation. Validation passed with 487 tests, 1 skip, and 70 subtests. Existing station launchers and Robot production behavior were not activated or changed; Phase 4 is ready.
2026-09-20: Manufacturing operating system Phase 2 completed. Manufacturing_System release v0.1.0-draft.1 supplies the shared validator; Project_Template main 859760d pins it and adds nested AI rules, CODEOWNERS and review evidence. A disposable repo passed contract, manifest and KiCad checks. Phase 3 shared-runner work is ready; no product or production repo was migrated.
2026-09-20: Manufacturing operating system Phase 1 completed at EmbeddedEra/Manufacturing_System commit aa2dd53. Schemas, state machines, Robot compatibility and event mapping, acceptance scenarios, gaps, and validation evidence are recorded; Phase 2 is ready. No live tester or ERP behavior changed.
2026-09-20: Angel approved the recommended manufacturing-operating-system defaults. Phase 1 started in private repo EmbeddedEra/Manufacturing_System; baseline edce42c defines draft governance, schemas, state machines, Robot compatibility evidence, acceptance scenarios, and the gap report without changing live production.
2026-09-20: Consolidated the planned product-repository, Universal Tester, firmware, fixture, ERP, prototype, parts-lifecycle, AI-governance, and traceability architecture into manufacturing-operating-system-plan. Phase 1 is definition and Robot reference mapping only; it makes no live production changes.
2026-09-08: wrote this note from PROJECT_TRAINING_GUIDE.md (reviewed 2026-08-17), AGENTS.md, host-api/ docs, and the production sources. Cow Tester at 2.5.0, UT002 firmware at 1.1.0.
2026-09-19: Decided: test stations get serials from a server-side tester API (reserved in blocks, MMYY#### per product, keyed by chip ID) and send every run to it; the ERP holds units and test results, the NAS the raw logs. Design: tester-serial-service. Phase A (server) is being built on the assistant sandbox branch.
2026-09-19: Tester API phase A built (Sonnet worker, assistant sandbox branch fb8be47, then fixes ad01c1b after a live sandbox-ERP run). Proven on the sandbox: reserve → assign → pre-coating FAIL/rework/PASS → post-coating PASS gives a Finished-goods unit with its full test history, BO-0001 1/27, raw events on disk. Found live: test templates need a description; test posts must be idempotent; completion needs stock allocated. Blocker for production: 177 parts are trackable (see mrp Open). Next: phase B (station client, brief ready in the job folder), Cloudflare service token, legacy serial import.
2026-09-19: Angel decided against a browser-based operator screen for now: stations keep running the Python program locally; only serials and results go to the server.
2026-09-19: Phase B (station client) built on Universal_Tester branch tester-server-client (7819ec6, pushed, not merged): server serials in reserved blocks, offline reserve, outbox, per-OS folders, logs copied out of the repo, Robot's duplicate allocator removed; 391 tests. Live-tested against the sandbox tester API + sandbox ERP: serial 09260004 went from assign to Finished goods with both test results; a chip-less ABORTED run was stored but kept out of the ERP. The live run found a race in the server's ERP sync (fixed, assistant sandbox d801067). Worker notes: the Codex run (gpt-6-astra) used two Claude Sonnet sub-runs to write and review, per Angel's 2026-09-18 preference (≈ $5.33 list-price estimate on the Claude allowance), and hit codex-worker's 1500 s timeout after the work was complete; see the ai-capacity ledger.
2026-09-19: Live on production: tester API behind tester.embeddedera.com (Cloudflare Access service token + per-station key), station angel-mac configured, 88 legacy serials imported, client merged (Universal_Tester PR #2). Then Angel asked for no repeated serials at all: the count is now one sequence across products (assistant 77a6bfd), so the first Robot of this batch is 09260036, and the ERP's cross-part uniqueness is back on. SOP-006 v1.7.
2026-09-19: First live boards found two problems. (1) The station had reserved its block minutes before the switch to one global sequence, so the first two Robots printed 09260001/09260002 — numbers the Cow Remote already used on 09-02. (2) The Remote station had called /reserve 34 times and burned 656 serials. Fixes: reserve now returns the serials a station already holds and issues only the shortfall (assistant a8c0505, 376 tests); all unused reservations released; the 4 diagnostic serials voided. Angel chose to scrap the two Robot serials and retest: ERP units deleted via build/2/delete-outputs/, both cow-robot rows marked void (never reissued), station told to delete its pool files and the two chip lines from its local registry. Care needed: my first void statement matched by serial alone and also voided the Cow Remote's real 09260001/09260002; restored from the legacy registry. Serial rows are per (product, serial) — always filter on both.
2026-09-20: Live final-test check found Cow Remote post-coating results arriving normally (HTTP 200) for legacy serials 09260015, 18 and 27–34. Their latest results were PASS, but the ERP mirror correctly remained pending because these imported serials have no server-recorded pre-coating PASS/unit. Added Grafana Quality dashboard ee-test-results: a read-only one-minute mirror of tester events/serials into the reports database, showing latest final result, full run history and the ERP pending reason without creating inventory.
2026-09-20: Angel defined the planned Robot-board failure workflow in robot-board-quarantine-rework-plan: every actual board FAIL goes to Quarantine; unreadable boards use an ESD bag, Citizen receipt, and external 2 x 1.5-inch Zebra QR label; a future Rework Mode controls repair, full retest, release, or user-confirmed scrap. Critical inventory correction: do not complete the full Robot BOM at first test, because coating, screw terminals, nRF, and other later materials are not yet installed. Inventory must be consumed by physical production stage. Planning only; no tester or ERP behavior changed.
2026-09-20: Refined that planned workflow for fixture seating/contact failures: preserve every first FAIL immediately, but put a plausible contact failure into Pending Retest and review it once at batch end after unclamp/reclamp. A full controlled-retest PASS returns to production while retaining the first FAIL for quality metrics; a second FAIL creates Quarantine paperwork. Unreadable boards still need an immediate temporary Hold ID so they cannot become physically anonymous. Safety failures bypass retest. Planning only.
2026-09-20: Superseded the planned end-of-batch failure review with a simpler same-run prompt while the board remains clamped: Reseat and retest once, Move to Quarantine, or Equipment/setup problem. A controlled retest repeats the complete current stage only; PASS continues immediately with both results retained, FAIL creates Quarantine paperwork, and the station cannot offer the next board until disposition is resolved. If a failed board leaves the fixture, it must first receive its Quarantine identity and bag label. Planning only.
2026-09-20: Angel removed the proposed one-retest limit and automatic Quarantine after a repeated FAIL. After every FAIL the same board remains active and the operator chooses Reseat and retest again, Move to Quarantine, or Equipment/setup problem; the prompt repeats without a software retry limit. Every attempt stays visible in history and first-pass/attempt metrics. Safety faults force power-off and acknowledgement before another attempt, but the operator still chooses disposition. Planning only.
2026-09-20: Simplified the planned production UI to Angel's exact flow after board identification: run test; PASS -> Test next board; FAIL -> only Retest or Quarantine; Retest loops on the same board; Quarantine saves the case, prints the Citizen receipt and Zebra bag label, then offers Test next board. Station/equipment errors stay outside the board-failure disposition screen. Planning only.
2026-09-20: Angel decided post-repair routing is automatic, not an operator choice. Rework Mode reads the stored failure stage, last accepted checkpoint, and repair: a repaired pretest failure that passes returns to Final Assembly; a repaired final-test failure that passes moves to Finished goods; a repair that invalidates an earlier checkpoint forces that stage and every later required test again. Missing/contradictory history blocks release rather than guessing. Planning only.
2026-09-20: The planned Citizen Quarantine receipt is also the physical rework traveler inside the ESD bag. It needs a large handwritten WORK PERFORMED area plus parts replaced, technician initials and date. Rework Mode requires that information to be entered electronically before PASS releases the board, Scrap records disposition, or another failed attempt returns it to Quarantine; No repair performed is explicit rather than a blank record. Planning only.
2026-09-20: Robot Quarantine/Rework implemented. Tester API d2eb23f is live with staged inventory, permanent label-only serials, append-only cases, automatic release routing and confirmed native Scrap. Cow Tester 2.7.0 is on Universal_Tester main 212cb20; 473 tests pass. A real sandbox run proved unreadable FAIL → Quarantine → chip link/work/PRECHECK release → FINAL/Finished goods and a separate Quarantine → Scrap/Destroyed path. BO-0001 coating is now stock-tracked and 27 mL allocated. The physical Mac still needs to pull main; Citizen/Zebra output and powered fixture flow remain bench checks.
2026-09-20: Existing failed Robots 09260042 and 09260059 were staged for the new flow without deleting history: live unfinished BO-0001 outputs 87 and 114 are in In test/OK, exact PRECHECK material was consumed, and their 3 and 4 historical test results are attached. Neither has a Quarantine case yet. Run each in PRECHECK; PASS returns it to final assembly, while FAIL shows the new Retest/Quarantine decision after the Mac has Cow Tester 2.7.0.
2026-10-13: Fresh Mac clone of Universal_Tester: Cow Robot station failed twice before testing. (1) New .cow-test-venv lacked pyobjc-framework-AVFoundation (camera-by-name), fixed by pip install -r requirements.txt into the venv. (2) camera_color_autotune.py re-learned colours from the old 26 Sep result and set color_profile_tuning.hardware_verification_required: true, so load_production_led_camera_settings refused; fixed by git restore of robot-led-camera-settings.json and latest_color_autotune.json. Autotune always needs a robot_led_camera_test.py verification run after it.
2026-10-13: Cow Robot FINAL aborted "green LED landmark is not visible": the station opened the Continuity Phone camera, not the microscope. camera_index_by_name indexed AVFoundation's list order (FaceTime, VMS800S, Phone) but OpenCV's macOS backend numbers cameras sorted by uniqueID (VMS800S=0, Phone=1, FaceTime=2), same in OpenCV 4.14 and 5.0; confirmed by grabbing a frame from each index. Fixed as Cow Robot 2.12.3 (sort by uniqueID), Universal_Tester PR #13, merged 7401cde. Every historic good run used index 0. tests.test_server_client has 3 failures on clean main on this Mac (pre-existing, not investigated).
2026-10-13: Post-coating on 2.12.3: 09260062 PASS; 09260061 FAIL then PASS; 09260060 FAIL x3 then PASS. All 9 events (incl. 2 aborts on 2.12.2) sat in the cow-robot outbox: the launcher had built .cow-test-venv from the first python3 on PATH, PlatformIO's (~/.platformio/penv), whose OpenSSL points at a missing CA file, so every upload failed CERTIFICATE_VERIFY_FAILED. Rebuilt the venv on /opt/homebrew/bin/python3 (3.14.6; needed setuptools installed first for --no-build-isolation); old venv kept as host-api/.cow-test-venv.platformio-bak. Server reachable again; the outbox sends on the next station start.
2026-10-13: Launcher fix on sandbox 75751f0: rather than choosing a Python in six launchers, server_client.http_transport (the only HTTPS call site) uses certifi's CA bundle when the interpreter has no CA store (not on Windows, which uses the system store), so existing broken venvs on any station fix themselves. Verified PlatformIO 3.11 reaches tester.embeddedera.com through it. Full suite has the same 4 failures on clean main (test_server_client x3, test_production_serialization x1), pre-existing on this Mac.
2026-10-13: BO-0001 (pk 2) is 27/27: 09260060/061/062 post-coating PASS (server needed a few 2-min retry passes while InvenTree finished async completions; no fix needed) and 09260059 post-coating PASS 20:40 (it had passed pre-coating 2026-09-26). BO-0001 cannot close cleanly: the 21 Paul Shop boards (09260036-058) were completed before per-stage consumption, so 21 boards x 92.15 = 1935.15 of material is still allocated. Angel decided: consume it on close; future builds auto-close only when nothing is left allocated, else page. Built in assistant branch tester-build-autoclose 193239c (worktree ~/assistant-autoclose, not pushed): _close_finished_builds runs each ERP pass, TESTER_CLOSE_CONSUME_LEFTOVER_BUILDS=2 approves BO-0001. Sandbox ERP run (fresh 20:43 copy) closed BO-0001 and consumed exactly 1935.15. Deploy (ff main + push, env line, restart tester-api) awaits Angel's go.
2026-10-13: Auto-close deployed: assistant main 193239c pushed, TESTER_CLOSE_CONSUME_LEFTOVER_BUILDS=2 added to ~/.config/tester-api/env (backup env.bak-20261013), tester-api restarted 20:54. BO-0001 closed on its own (close:2 done): Complete 27/27, consumed 2488.05, allocated 0, matching the sandbox run. The env line is now inert (close:2 is done and never revisited); drop it whenever the env is next edited. SOP-006 1.9 published: the tester closes clean builds and pages otherwise.
2026-10-13: Launcher fix merged to Universal_Tester main (PR #14, 02ecd56, Cow Robot 2.12.4 / Cow Remote 3.14.1) at Angel's call, before a hardware bench test. Mac clone is on main and its .cow-test-venv reinstalled (certifi, AVFoundation present).
2026-10-14: Tester API cleanup (option 1 of the API/custom-code plan) started. Slice 1 on sandbox e61e605: universal_tester.discovery (host API 1.7.0) replaces 6 copies of the tester port search and 4 of the Siglent VISA search; stations import it under their old names. Fixed a real bug: UniversalTester()._discover_port matched USB ID 0483:5740 alone, which a Cow Robot board shares, so the client could open the board under test. Decided: Python stays the engine (no Rust/Flutter rewrite); the screen is a cockpit-style web page in a pywebview window; no home-made script language, test plans stay TOML (Robot Framework only if customers must write tests); copy git-ai-commit's GitHub Actions release pipeline for per-OS installers.
2026-10-14: API cleanup slice 2 on sandbox f729145: universal_tester.production.fixture_identity (host API 1.8.0) replaces 3 record builders, 2 verify-or-engrave copies and the fixture-station parser. Engraved bytes pinned by test to the 2.12.4/3.14.1 output for both fixtures.
2026-10-14: API cleanup slice 3 on sandbox 2ba684f: universal_tester.production.publish_test_run (host API 1.9.0) replaces both stations' publish_run_to_server bodies (Robot keeps required=True, so a run that could not be queued still stops the station). The production README now points new fixtures at it.
2026-10-14: Serial month checked, no bug. The station learns MMYY from the tester server's Date header (never the PC clock), treats a past-month pool as stale, reserves a fresh block and discards the old one. Cow Robot's pool rolled to 1026 at 20:28 on 2026-10-13, its first server contact after the TLS fix. Cow Remote's pool still holds 0926 serials only because it has not run since 2026-09-20; its next online launch replaces them. Known edge, by design: a station that cannot reach the server on the first launch of a month does not know the month and keeps using its old pool.
2026-10-14: API cleanup slice 4 on sandbox 07ced99 (host API 1.10.0): TestStation is the front door. find_station_hardware() names every missing device at once; TestStation.discover(fixture=..., supply_limits=...); connect() verifies the FixtureRecord and shuts down on the wrong fixture; open_servos() powers servos from SPD CH2 and shutdown parks them before outputs go off. README section "The station front door". Both stations use find_station_hardware; Cow Remote now reports all missing devices together.
2026-10-14: API cleanup slice 4b on sandbox: 3982086 (host API 1.11.0) gives TestStation uut_channels (only DUT channels are switched/confirmed off; both stations keep SPD CH3 on, Robot's status LED runs from it) and splits shutdown into power_off_uut() (board power off except the servo channel, servos park, all DUT channels confirmed off, safe routing; devices stay open for labels/LED/chime) and close(). Cow Remote 3.14.6 and Cow Robot 2.12.9 (9b4695c) now connect, open servos and make the board safe through it. Robot also confirms CH1/CH2 off at connect (was assumed).
2026-10-14: Production PRECHECK batch on Cow Robot 2.12.10 (hotfix branch hotfix/ch1-rail-settle 7175fee, not yet merged): CH1 rail check now waits for the rail (polls ADC2 0.1 s, up to 2 s); every run settled in 0.00-0.001 s and passed. History before the fix: 8 of 23 first-runs-after-idle read 2.1-2.6 V vs 12 of 190 back-to-back. Results so far: 10260002 (after reseat; first run 1.3 mA = not seated), 10260003, 10260004 PRECHECK_PASS. 10260001 (2041377D3045) and 10260005 (205D32945246) fail identically: normal 15 mA, DFU OK, Robot_4.5.0 flashed and verified, then the app never enumerates CDC; on retest the board draws ~1.9 mA and servo DFU entry fails. Same signature as 1.88/1.94 mA PRECHECK_FAILs on 2026-09-19. Not the USB path (3 passes through it). Suspect board or production-firmware behaviour on those boards; recover one with the ST-Link (station option 5) to see its state. Diagnostic rule learned: initial DMM current ~1-2 mA means the board's MCU is not powered (fixture parts only) -> reseat; 10-25 mA is normal (150 of 165 such runs found the chip).
2026-10-14: ROOT CAUSE of today's 'bad boards' (10260001 2041377D3045, 10260005 205D32945246, 10260006 203D377D3045): the 2026-09-19 option-byte brick again. Probe of 10260006 in the fixture (no flash): steady 1.95 mA on CH1, no CDC and no DFU on USB. Evidence: (1) post_flash_first_boot_cdc_seen is False on ALL 13 flashed runs since the 2026-09-19 fix (dabfa9d), passes included, so the fix's 'wait for the first boot' never engaged; (2) the station ends flashing with dfu-util -e (detach_stm32_dfu, output discarded, check=False), which does not leave the ST ROM bootloader (ST community: 'can't detach'; use DfuSe :leave or a reset), so the firmware's real first boot is the station's next USB power-on; (3) Cow_Robot_Board main.c calls OB_Configure() right after HAL_Init(), before clocks, with BOR still off (factory), so a not-yet-settled supply at power-on corrupts FLASH_OPTR. No STM32CubeProgrammer on the Mac, so the dfu-util path is used. The boards are good: all six 2026-09-19 bricks were recovered over SWD and passed. Cow Robot 2.12.12 (9d60c49, hotfix branch) refuses to flash a board whose firmware has not run this attempt (ABORTED_SETUP before any serial; override COW_ALLOW_BLANK_FLASH=1); re-flashing a board that already ran its firmware is safe because OB_Configure then writes nothing. Stale station process closed so the next launch runs 2.12.12.
2026-10-14: FIX for the post-flash bricks, Cow Robot 2.13.0 / host API 1.7.0 (9d1c812, hotfix branch the station runs). After flashing (now with detach=False: a detached STM32 bootloader sits in appIDLE and stops answering uploads), the station writes FLASH_OPTR over DFU to exactly Cow_Robot_Board OB_GetDefaults() = value 0x3E5FFF00 under mask 0x3F7FFF00 (BOR on L3/L3, nRST_* 1, IWDG/WWDG sw, nSWAP_BANK 1, DUAL_BANK 0, RAM parity off, nBOOT_SEL 0, nBOOT1 1, nBOOT0 1, NRST_MODE 3, IRHEN 1; note ST names OB_IRH_DISABLE = the SET bit). DFU option bytes: alt 1 at 0x1FFF7800, 128 bytes, value/complement word pairs; dfu-util on macOS names every alt 'Internal Flash' so :force is needed; a write ends with a get_status error because the bootloader reloads and resets (expected). Board 10260006 (203D377D3045) recovered with the ST-Link and OpenOCD 0.12 (brew open-ocd, free; STM32CubeProgrammer is NOT required): stm32l4x unlock 0 then stm32l4x option_load 0 (RDP back to 0xAA, mass erase), stm32l4x option_write 0 0x20 0x3E5FFF00 0x3F7FFF00 + option_load, program Robot_4.5.0.bin 0x08000000 verify. Gotchas: an OpenOCD option_write together with mass_erase did not persist (verify with mdw 0x1FFF7800); a G0 decides 'flash empty -> bootloader' only at POR/OBL, so after a mass erase use option_load, not reset; leaving the core halted makes the DFU device report serial UNKNOWN. Proof on that board made factory-fresh (blank, OPTR 0xFFFFFEAA verified over SWD): the station's own flash_robot_firmware + preset_stm32_optr flashed/verified 4.5.0 and left OPTR 0xFEDFFFAA (DFU and SWD agree).
2026-10-14 19:25: FIRST BOOT CONFIRMED on board 10260006 (203D377D3045) after the 2.13.0 preset: SWD lead removed, USB replugged, the firmware came up on its own as /dev/cu.usbmodem203D377D30451 and answered IDENTIFY -> COW_ROBOT_V4.5.0. First time a freshly flashed Robot has enumerated on its first boot since 2026-09-19. Board 10260006 is good.
2026-10-14: Recovered over SWD with OpenOCD: 10260005 (205D32945246) and 10260001 (2041377D3045), both found at FLASH_OPTR 0xFFFFFECD; now 0xFEDFFFAA stored and Robot 4.5.0 verified. Gotcha: stm32l4x unlock 0 sometimes does not take on the first run (10260001, and 10260006 earlier): always reread 0x1FFF7800 and repeat unlock + option_load until it reads ...aa. All three of today's bricked boards are recovered; first boot seen on 10260006, the other two still to be booted (SWD lead off, USB replugged) and PRECHECKed.
2026-10-14 20:15: 10260001 (2041377D3045) first boot confirmed after recovery: came up on USB on its own and answered IDENTIFY -> COW_ROBOT_V4.5.0.