Quarantine/rework flow: make the Citizen receipt (and bag label) print optional, a prompt with Skip; never block disposition on printing (owner 2026-11-07). Station code, test on the station.
HANDOFF 2026-10-22 (production firmware switch): DONE + pushed to main: Cow Robot 3.6.0 (12436ed) flashes Robot_4.8.4.bin; Cow Remote 3.16.0 (749775b) flashes Cow_Remote_6.15.0.bin. Both hashes match the Cow Flasher site main Firmware/. Checks passed: unittest 594, only 9 pre-existing failures (identical on old main). Open: (1) neither change run on hardware: on the Mac git pull main, then one robot PRECHECK+FINAL (IDENTIFY 4.8.4, nRF delivery >=90%, LEDs) and one remote full run (IDENTIFY 6.15.0 after final flash); (2) Phase 4 release package products/cow_robot/release still pins 4.5.0 (not wired to station); (3) remote 6.2.4 IDENTIFY special case in remote-board-test.py (~1243) is now dead code, can go after (1) passes. Next action: bench run (1).
Station side of 'serial only on PASS' (decided 2026-10-21, see tester-quarantine-no-serial-server-plan): in cow-board-test.py remove ensure_run_human_serial before flashing (~line 4040) and in the fail cleanup (~4440); assign after a PASS where the Brother label prints; Brother sticker PASS only; send quarantine without serial; Citizen receipt + Zebra label print QID/chip/failure (print_quarantine_documents, QuarantineWorkflow.failure drops the 'no permanent serial' hold); drop reserve_label_serial for new boards; rework PASS assigns and links the serial. Start AFTER the server accepts serial-less cases on the sandbox. Then update tests (test_quarantine_workflow, test_cow_led_camera_production).
2026-11-07 Pushed main to origin (749775b..ec32f0b): 3.7.0 fleet colours, 3.8.0 red/orange hue gap, 3.8.1 launch calibration prompt only when a stage is unverified, plus 11-07 FINAL calibration data (verified_result 20261107T211746Z). Owner finished all boards for the day.
2026-11-07 Board 203F32935246 still failed ORANGE colour under the fleet profile (orange 8.85 deg = reddest on record, margin 0.17). Added Cow Robot 3.8.0: RED/ORANGE colour also passes when the board's orange is >= 3.0 deg hue above its own red (history min 3.9) with other gates passed and frames read as commanded. Replay 157 tests: 14 retest-passed colour fails rescued, 0 swapped-LED sims. Committed, not pushed; station restart needed.
2026-11-07 Camera RED/ORANGE colour fails = single-board profiles, not drift. Camera never picked the wrong colour; boards fell under min_color_margin 0.25 because LED bin/coating varies (coated red hue 2.8-7.7 deg today vs 3.8+-0.6 on 10-21; uncoated 6.4). Fix (uncommitted in Universal_Tester): auto-calibrate (C/F) now learns from every board in all-board-tests.jsonl since the exposure lock (POPULATION_SINCE 2026-10-21) plus the known-good capture, and re-learns once when the active profile came from one board. Proof: trained on 10-21 history + 2:20 PM capture, today's 53 unseen coated boards: colour fails 12 -> 1, wrong-colour frames accepted 20 -> 0; swapped-LED replay wrongly passed 9/906 -> 1/906. Holdout on coated-only 10-21 data made it worse (24) - coating spread needs both stages. Tests 615, only the 4 pre-existing serial/folder failures. Running station needs restart + F calibration to use it.
2026-11-07 Owner: final test must flash the latest robot firmware published on therealcow.embeddedera.com. Latest there = Robot_4.8.4.bin (recommended, 'Pairing-mode release', sha 8920805c...6f91, flasher main). Station already pins it since Cow Robot 3.6.0 (12436ed); Mac checkout was 2 behind, pulled to 749775b; 36 manifest/LED tests OK in .cow-test-venv. Station running since 11:29 was started on 3.5.0 (flashes 4.5.0) and needs a restart.
2026-10-22 Cow Remote 3.16.0 (749775b, pushed to main): production remote is now Cow_Remote_6.15.0.bin, the site's recommended remote (sha d1a2e3aa...7114 = flasher main Firmware/; IDENTIFY COW_REMOTE_V6.15.0, USB 'The Real Cow Remote'). firmware_manifest.py cow_remote rule now (?:Cow_)?Remote_x.y.z.bin so the site filename is kept. Test firmware 6.14.0 unchanged. Tests 594, only the 9 pre-existing Linux failures. NOT yet run on a board.
2026-10-22 Cow Robot 3.6.0 (12436ed, pushed to main): production flash is now Robot_4.8.4.bin, the pairing release the Cow Flasher site recommends (sha 8920805c...6f91 = site flasher main Firmware/ = robot repo 17a0f5b build). Unpaired boards stay on legacy V1 radio (COWAD/0x4C), so the nRF test is unchanged; binary has LED_CURRENT_MODE/LED_TEST/GPIO_STATUS/BOOT_LOADER, IDENTIFY COW_ROBOT_V4.8.4. Phase 4 release package (products/cow_robot/release) left at 4.5.0. Tests: 594, only the 9 pre-existing Linux failures. NOT yet run on a board: prove with one PRECHECK + FINAL (IDENTIFY 4.8.4, nRF delivery %, LEDs).
2026-10-21 ~25 boards at FINAL PASS but the build order shows 1 finished: read-only check of the tester DB and InvenTree from the Mac. Station is fine (outbox empty, nothing rejected, all events accepted). Cause is on the server/ERP: (1) the 3 open part-181 build orders are full (9 outputs each), so 36 serial_assigned events wait ('create or issue another'); (2) two material-consumption operations stuck 'started' (stock items 374 = serial 10260007 on build 3, 408 = serial 10260016 on build 5, both quarantined) block every other unit on builds BO-0002 and BO-0003. Brief for the server session: tester-erp-backlog-2026-10-21. Nothing on the server was changed.
2026-10-21 Cow Robot 3.5.0 nRF radio fails diagnosed: 82 FINAL runs with nRF data, 10 radio fails, 8 on 10-21. Eight are 85-89% delivery (limit 90%) and 6 of 7 passed on the next attempt; two are real (board 205F32945246: 6% then 26%, replies 5/0/0/0 and 18/3/0/0 per 5 s phase, link died during the test; inspect its radio/antenna/solder). Cause of the marginal ones is the test, not voltage or hardware: from the detail logs, replies per 5 s phase on 32 passing runs are 17.4 / 19.7 / 20.0 / 19.7, so the first ~0.5 s after power-up has no link, and counting from the first packet puts a normal board at ~94% +/-3 (binomial, 82 packets) against a 90% limit. Fix: wait_for_nrf_link (neutral packets until first reply, 3 s timeout) before the test clock starts; limit unchanged; nrf_link_up_seconds recorded. Expect steady-state ~96%. Not run on a board yet; check nrf_link_up_seconds in the next records and the delivery percentages.
2026-10-21 Cow Robot 3.4.0: main menu 0 = PRECHECK of an already-coated board (post-coating camera settings, banner 'PRECHECK - COATED BOARD'). Trigger: chip 2076327D5246 was coated before any pre-test, FINAL refused it. Serial 09260063 checked: label-only reservation from 2026-09-26 (unreadable-chip board, never linked, no quarantine case), not tied to that chip; Angel is fine assigning a new serial (a pass assigns it). Note a coated board's PRECHECK is filed as stage pre_coating on the server.
2026-10-21 Cow Robot 3.3.1: failure menu = Retest / Quarantine / Next board / Main menu (Main menu returns to the mode menu, no case/receipt/serial). Observed: a FINAL on chip 2076327D5246 raised RuntimeError 'Final test requires a passing PRECHECK' (cow-board-test.py ~545), which is not a ProductionSetupAborted, so it is recorded as a board FAIL and offers Quarantine. Open question for Angel: should that be a setup abort instead (no quarantine offer)?
2026-10-21 Cow Robot 3.3.0: FINAL no longer prints the Brother serial label (no printer connect, no REPRINT/KEEP prompt); the serial sticker goes on at PRECHECK only, bag label unchanged. FINAL plan = 14 steps (13 without the bag label); label_print_status NOT_REQUIRED_FINAL. A board reaching FINAL with no serial still gets one assigned on PASS (needed for the bag label).
2026-10-21 Cow Robot 3.2.2: removed the '... checking the LEDs (about N s)' progress lines added in 3.2.1; Angel dislikes extra comments on the operator screen and wants PRECHECK and FINAL screens identical. The NOTE line on a failed attempt with fewer steps than planned stays. Consequence: the ~21 s radio/limit/UART block (steps 7-13 print together) is silent again; if that is raised, the fix should be a progress bar, not text.
2026-10-21 Cow Robot 3.2.1: FINAL 'stuck between 7 and 13' and '1 of 15 but finished at 13' diagnosed from detail logs. Steps 7-12 (nRF radio, 2 limit pogos, 3 UARTs) are one ~21 s block (radio_uart_and_current) whose results print together at its end, and the quiet screen hides measurements, so the screen sat still for 21 s. The 13-of-15: attempts 16:50 and 16:52 failed DUT LEDS (orange colour, coated board before the F calibration) but still ran steps 7-13; the two label steps (14 serial, 15 bag) are PASS-only, so a failed FINAL reports 13 of 15 by design. Fix: ' ... doing X (about N s)' lines at the firmware (7 s), LED camera (12 s) and radio (20 s) blocks, shown by the quiet screen; a NOTE line on a failed attempt that reported fewer steps than planned. The 16:57 FINAL passed 15/15 in 63 s. Not run on a board yet. If it truly hangs (not just silent), check the detail log tail of that attempt.
2026-10-21 Cow Robot 3.2.0 (fast-forwarded into the main checkout, active after a station restart): separate LED camera profiles per stage. Evidence: coated board 207932A352 failed FINAL ORANGE colour 2/2: orange red_fraction 0.628 (uncoated PRECHECK passes 0.587, saved profile 0.559), weighted hue 10.5 vs 12.3, orange-vs-red separation 0.03 vs required 0.25 (uncoated passes 0.65); red also shifts redder. Now PRECHECK uses robot-led-camera-settings.json, FINAL uses robot-led-camera-settings-post.json (created on first FINAL use as a copy of the pre file with hardware_verification_required=true, so FINAL is blocked until calibrated). Calibrate: menu 3 or launch prompt, C = bare board, F = coated board (known-good). Not run on hardware yet; the post file does not exist until the first FINAL attempt or F calibration.
2026-10-21 Cow Robot 3.1.1: Next board (3.1.0) asked a second 'Test next board / Return to menu' question; now returns 'skip' and station_loop starts the next test at once like Retest. Angel: 'that button should just start testing the next board'.
2026-10-21 Cow Robot 3.1.0 (fast-forwarded into the main checkout, active after a station restart): a failed board offers Retest / Quarantine / Next board. Next board = no quarantine case, no receipt, no serial; run stays logged/sent; board can be retested or quarantined later (it is an undispositioned failed board until then, so work-order reconciliation will show it). Opt-in via QuarantineWorkflow.failure(allow_next=True), Robot only; Cow Remote unchanged. Supersedes agreed rule 2 'only Retest or Quarantine' (memory universal-tester-pickup updated).
2026-10-21 Cow Robot LED camera: board 206732965246 failed ORANGE 'light' 3 of 3 (PRECHECK) with normal LED current (orange +1.38 mA vs median 1.39) and correct colour/position: the orange LED is optically dim because D15 sits rotated/offset (visible in camera-evidence; D15 is crooked on many boards, orange looks like a different, wider package than red/green). Lit-minus-OFF bright-fraction rise was 0.114-0.124 vs required 0.15; 135 passing orange states have min 0.181, median 0.256; a dead LED rises ~0. Auto-calibrate cannot absorb it by design (the tune refuses a capture whose light gate failed). Set thresholds.min_target_bright_fraction_rise 0.15 -> 0.08 in robot-led-camera-settings.json (Angel: indicators need not be bright); offline re-evaluation: board passes, simulated dead orange LED still fails. Revert = set it back to 0.15. OPEN: why D15 sits crooked (footprint/pad length/paste?) - Cow_Robot_Board files are not on the Mac; compare the land pattern with the LED datasheet and check whether the orange part differs from red/green.
2026-10-21 Cow Robot 3.0.0 (commit 5ad0ddd, fast-forwarded into the main checkout, active after a station restart): serial only on PASS. Trigger: a PRECHECK_FAIL on a board whose DFU never enumerated (chip unreadable) prompted 'existing permanent SN', reserved serial 10260045 and printed a Brother sticker. Removed the assignment in identify_current_board, before power-up/flash, and in the fail cleanup; Brother label skipped on fail (NOT_PRINTED_TEST_FAILED). INTERIM: choosing Quarantine assigns the serial then (QuarantineWorkflow.failure(assign_serial=...)) because the server still needs a serial to open a case; remove when the server plan lands. Cow Remote station untouched (shares QuarantineWorkflow, no assign_serial passed). Tests: 3 workflow tests + 1 timing test fail on the old code, pass now.
2026-10-21 Cow Robot 2.14.0 (branch camera-autocal, fast-forwarded into hotfix/ch1-rail-settle as 33113af, NOT yet run on a board): (1) main menu 3 = CALIBRATE CAMERA + LEDs, and the same prompt at station launch (C calibrate / S skip, shows age of last verification); if the first verify already passes nothing is changed, else tune + verify. (2) Exposure lock: capture.lock in robot-led-camera-settings.json, applied by USBCamera.open via uvc-util (universal_tester/optical_camera_lock.py); the first CALIBRATE adds it (ensure_lock) and sets hardware_verification_required so production stays blocked until a verify passes under it. Measured on the VMS800S: OpenCV/AVFoundation cannot read or set any exposure/WB property (all -1); uvc-util (MIT, github.com/jtfrey/uvc-util, built with clang -o ~/.local/bin/uvc-util -framework IOKit -framework Foundation uvc-util.m UVCController.m UVCType.m UVCValue.m in ~/src/uvc-util) works: auto-exposure-mode 1=manual 8=auto, exposure-time-abs 1-2500 (100 us units), brightness rises to 250 then plateaus, so lock = manual @ 250. The auto-mode value it reports (156) is NOT the real exposure (auto frames were ~2.3x brighter than manual 156), so values cannot be learned from auto. white-balance-temp is accepted but ignored, so WB cannot be pinned; focus-abs is a stub (min=max=0). If uvc-util is missing a configured lock stops the LED test with an error (new Mac: rebuild it). Pre-existing unrelated test failures on this Mac: 4 (serialization/server_client folder-state tests), same on 2.13.0. TO VERIFY: first launch, press C with a known-good board, then check that the next two sessions pass verify without re-tuning.
2026-10-21 Cow Robot camera "recalibrate every session" diagnosed (read-only, station was mid-run, nothing edited). Evidence from robot-led-camera-tests: on 09-26, 10-15 and 10-21 the FIRST verify failed on ONE colour gate (red/orange colour or green activation) and AUTO-TUNE then VERIFY passed minutes later. Software board alignment was healthy every time (scale 0.99-1.0, rotation <1 deg, 43-65 inliers), so position needs no recalibration. The drift is exposure/white balance: OFF-baseline brightness swings 160-255 and LED brightness delta 55-98 between sessions; optical_camera.py only READS exposure/auto-exposure/auto-WB (CameraCapabilities) and never sets them, so the VMS800S (UVC, resets Mirror at power-up) auto-exposes differently each launch. PROPOSED: (1) try to lock manual exposure + WB in USBCamera.open (test whether AVFoundation honours CAP_PROP_AUTO_EXPOSURE/EXPOSURE/WB on the VMS800S; else UVC control via a maintained tool), (2) on station launch run VERIFY and, only if it fails on colour/activation, offer AUTO-CALIBRATE (menu 6) with a known-good board, (3) one-key CALIBRATE on the main menu. Not built yet; needs the camera free.
2026-10-06 Cow Finger 003 test plan: test bare reflowed boards before wiring, via a printed spring 'bite' clip (5 x P75 cone pins up onto J1's 0.5 mm holes at x3.95, registered in H1/H3, stop wall) wired to J5 of the existing Cow Remote fixture (GPA0-3 as pulled-up inputs). Test = idle-high, per-line short check, guided SW1-SW6 presses where only that line may move, label on pass. SW3/SW4 and SW5/SW6 are paralleled, so operator order is what proves each. No off-the-shelf 3.5 mm pitch pogo clip exists, so custom. Clip P01 CAD (build123d) in Universal_Tester fixture-cow-finger/clip-p01, geometry checks pass, NOT printed. 6-clip multi-board needs a new fixture PCB (2x MCP23017 + 24LC64), deferred until single clip proves out.
2026-09-26 MERGED to main: Universal_Tester PR #11 as d211a3e (Universal Tester Station launcher + Cow Robot 2.7.1 -> 2.12.2). Angel confirmed all of today's boards good in production. Added since the earlier log lines: 2.10.0 quiet operator screen (results/prompts/failures only, every line to cow-robot/details/
2026-09-26 Cow Robot board 2035377C3045 (SN 09260061) PRECHECK PASS 14:2x (all 12 steps, 37.8 s) after re-learning LED colours. Why it had failed 'RED: color' while visibly good: in the microscope's POWER-UP state the whole scene reads brighter (unlit LED bright fraction 0.67-0.72 vs 0.42-0.62 on 09-20) and RED paler (red_fraction 0.59-0.62 vs profile 0.678), so red-vs-orange margin fell to 0.10 (< 0.25). Fix = VERIFY -> AUTO-TUNE -> VERIFY on the power-up camera; RED profile now 0.54, margin 1.06; committed 18650a7 with its result.json files (settings must reference a tracked verified_result). Also found: robot_led_camera_test.py used LED_TEST, whose OFF left the green POW LED lit on 2035377C3045 (GREEN then showed no rise over OFF); 2.9.1 (b297f03) switches it to LED_CURRENT_MODE like production and restores NORMAL. 2.9.0 (cf7b7af) adds CAMERA/LED SETUP option 6 AUTO-CALIBRATE (verify -> tune from that capture -> verify, production software alignment, no preview); not yet run on hardware. Firmware-speed review: BOOT_LOADER CDC command exists in production (main.c:899) and test firmware, usable to enter DFU for flashing without servos, but cannot test the BOOT button (PA14 is SWCLK, not readable). Phase 4 manufacturing-test firmware T.4.0.0 is built but lacks LED_CURRENT_MODE/LED_TEST/GPIO_STATUS and its IDENTIFY 'COW_ROBOT_VT.4.0.0' fails the station regex; station not wired to Phase 4. Unpushed on universal-station: 0862cea a19152d bca8669 fd35fd1 cf7b7af b297f03 18650a7.
2026-09-26 Cow Robot 2.8.1 verified on hardware (13:48, board 203B377D3045 / SN 09260059): CH1 rail 3.339 V PASS, initial current 12.298 mA PASS with the meter held on 200 mA, camera auto-aligned (45%) and OFF/RED/ORANGE/GREEN all PASS, RESET PASS; BOOT SWITCH FAIL (DFU did not enumerate in 30 s). 203B377D3045 has failed BOOT on 5 of 5 runs ever (09-19 x4, 09-26 x1) while the fixture overall fails BOOT 21/150 and the other BOOT0-experiment board 2042377D3045 now passes. It was used in the 09-19 BOOT0 experiment that set nBOOT_SEL=1 nBOOT0=0 (ROM ignores the BOOT pin); nothing in the log says it was put back. Next: wire the ST-Link, station menu 5 RECOVER (reads option bytes and names any field that differs from known-good 206E377B3045; mass-erases only on an explicit yes), REMOVE the SWD lead, retest.
2026-09-26 Cow Robot known-good board 2054377B3045 (SN 09260060) PRECHECK PASS at 13:45 on 2.8.0 after four attempts. What was wrong, none of it the board: (1) LED camera: the VMS800S Mirror setting RESETS AT POWER-UP, so the correction above ('set Mirror back') was wrong; Cow Robot 2.8.0 (commit a19152d, universal-station) aligns both orientations, keeps the one that matches the reference and records led_camera_mirror; landmark fallback decides from GREEN left of ORANGE; camera_live_alignment.py now resolves the camera by NAME (it used fixed index 0). (2) CAMERA/LED SETUP option 4 AUTO-TUNE was run three times (07:22, 12:15, 13:36) and each run sets color_profile_tuning.hardware_verification_required=true, locking production; it only re-derived profiles from the same 2026-09-20 result, so settings were restored to git HEAD each time. Do not use option 4 to fix alignment. (3) CH1 POWER RAIL read 2.09 V (good boards 3.295 V): SDM3055 idle on AUTO sat on a small range and its shunt dropped ~1.2 V at 14 mA; proven by the next attempt reading 3.336 V once the meter was on 20 mA. 2.8.1 (commit bca8669) holds the 200 mA range (SCPI SENSe:CURRent:DC:RANGe 0.2, verified on the meter, auto=0, no error) through rail + initial current, then restores AUTO. (4) 'Camera returned no frame' on open: 2.8.1 waits up to 3 s for the first frame. OPEN QUESTION: PRECHECK initial-current floor 12.0 mA is marginal for an ALREADY-FLASHED nRF-less board: this one read 11.59, 11.59, 12.18 mA (LED-off current 11.77 mA); earlier flashed retests read 12.2-12.6; dead boards read 0-2 mA. 2.8.1 not yet run on hardware; 0862cea/a19152d/bca8669 not pushed.
2026-09-26 Cow Robot camera 'cannot find the board' with a sharp, focused picture: the VMS800S microscope's own Mirror setting had been toggled. The station mirrors every frame (pillow_imageops_mirror) because the camera USED to send a mirrored image (silkscreen reads backwards in raw frames, e.g. robot-led-camera-tests/20260920T183939Z/*_raw.jpg); with the camera now sending a normal image the station's flip made it backwards, ORB registration got 4-5/55 inliers and told the operator to 'rotate board -64 deg', and the LED-landmark fallback fails too because green/orange swap sides. Proof: today's frame flipped back registers 52/131 valid. Fix: set the microscope Mirror back so the silkscreen reads BACKWARDS on its screen. Cow Robot 2.7.1 (commit on universal-station) now retries alignment flipped and raises CameraMirrorError with that instruction. Side effect found: camera_color_autotune was run at 07:22 while flipped and set color_profile_tuning.hardware_verification_required=true, locking the station; restored robot-led-camera-settings.json to the 2026-09-20 hardware-verified version (autotune copy kept in the session scratchpad). Do not run the colour autotune to fix an alignment problem.
2026-09-20 Cow Robot station: (1) LED 'light' gate now also passes on a rise of >=0.15 above the same LED's own OFF bright fraction (optical_decision.evaluate_led_state; good boards rise 0.24-0.50, dead ~0) because absolute fill varies 0.81-1.00 between good boards/sessions; brightness delta now uses the darkest OFF frame since the green heartbeat blinks during OFF and zeroed paired deltas. camera_color_autotune sets hardware_verification_required, which blocks the production station until examples/robot_led_camera_test.py passes; that script does not power the DUT, USB1 must be on first. (2) FINAL initial-current low window widened 14-18 -> 14-22 mA: prechecked boards with nRF fitted read 19.6-20.1 mA at power-up (2056377B3045, 2068377B3045) then 34 mA in packet RX with 92% delivery; August boards read 34 mA at the same stage on the same 4.5.0, cause of the difference unknown (radio apparently not listening at boot). Dead radio is still caught by the 30-45 mA packet-RX gate.
2026-09-20 Labels can now be switched off per station, menu option LABEL PRINTING (Cow Robot option 6, Cow Remote option 3; commit ef99738 on main). Two labels exist on both stations: the SERIAL NUMBER label (Brother PT-P900, 18 mm laminated tape) that goes on the BOARD, and the BAG label (Zebra GX430t over the network) that goes on the finished-goods bag. The Citizen CT-S801 only prints rework receipts on failures and is not part of the toggle. Choice is saved in label-options.json beside the logs, per product, and survives a relaunch. Switching a label off ALSO skips connecting its printer - that is the point, because label_printer.connect() / bag_label_printer.verify_connection() are unguarded and an unreachable printer aborts the whole run, so a station could not test at all without both printers. Three rules kept because labels are traceability records: a skip records DISABLED_BY_OPERATOR rather than looking un-attempted; it never fails a board (passed = test_passed and shutdown_ok and label_ok, so a deliberate skip leaves label_ok alone, while a label that was meant to print and could not still fails); and the setting is printed in orange on the station banner at launch AND before every board so it cannot be forgotten. A corrupt settings file falls back to PRINTING everything, never to skipping.
2026-09-20 Cow Remote log merge APPLIED and verified. 211 test records (211 unique run_ids, zero duplicates, chronological, 2026-08-16 -> 2026-09-20, 108 PASS / 103 FAIL), 64 serials, 64 label records, 40 bag labels, 13 rework actions, 4 button diagnostics, 2 diagnostics - every file parses clean. Backup at ~/Library/Application Support/EmbeddedEra/UniversalTester/cow-remote.backup-20260920-113035 and MIGRATED_FROM.txt written so the station will not try again. Verified through the station's OWN functions, not by eye: find_passing_remote_pretest('344C39463335') returns run 20260919T161929.824962-0400, PRETEST / INITIAL_PRE_COATING / PASS; has_passing_remote_pretest True; lookup_production_serial 09260018. enforce_final_pretest needs both of those, so FINAL test on that board will now pass its pretest gate.
2026-09-20 All sandbox work merged to main and pushed (merge commit 0a36349, --no-ff so the 13 commits stay individually visible; main went 65 -> 79 commits, nothing left behind). sandbox still exists at cd9cc3a and is fully contained in main. Covers: Cow Robot 91.9 s -> ~46 s, the first-boot option-byte brick fix, SWD recovery (menu option 5 + standalone module), the bootloader-vs-dead diagnostic, the shared CubeProgrammer flash path that Cow Remote also gets, and the per-file log migration fix. Tests 426 pass, 1 pre-existing unrelated failure (test_server_client.py ConfigTests::test_file_values_and_defaults) that also fails on the pre-merge main - NOT introduced here. STILL NOT DONE, carry forward: (1) no board has been through a FULL station cycle with camera and printers since the speed work, because macOS TCC denies camera access outside the station process - run one known-good board and check logged led_current_*_average_a against history; (2) DONE 2026-09-20 11:30 - the cow-remote log merge was applied; (3) the BOOT0 pull-down question is still open, the firmware reads the pin and takes its BOOT_LOADER backdoor when it is high, which is the best explanation for the original intermittent CDC failures.
2026-09-20 Cow Remote "this board was not tested" when it HAD been: the station was reading a different folder, not missing the record. migrate_legacy_logs refused outright when the data folder was not empty ("data folder not empty" blocker). Something wrote manual-rework-actions.jsonl into ~/Library/Application Support/EmbeddedEra/UniversalTester/cow-remote BEFORE the station first ran, so the one-time migration was skipped PERMANENTLY and the station started a fresh all-board-tests.jsonl. 209 test records and 63 serial assignments stayed in host-api/examples/log-cow-remote, invisible to find_passing_remote_pretest which reads only PROFILE.paths.jsonl_log (the data folder). enforce_final_pretest then raised "Final test requires a passing PRETEST" for board 344C39463335 whose PRETEST passed 2026-09-19T16:19:29. Tell-tale: the cow-remote data folder had NO MIGRATED_FROM.txt while cow-robot did. FIX (sandbox): migration is now per-file - copy every legacy file the target lacks, never overwrite one it has, and NAME the skipped ones in the return value, the marker and a start-up warning, because a split history is what makes a tested board look untested. REPAIR TOOL for a station already split: host-api/examples/merge-split-station-logs.py --product cow-remote (dry run; --apply to write, backs the data folder up first, never touches the legacy folder). Merge rules: test records on run_id, serials on board_identity with the data folder winning, append-only logs keep EVERY record including duplicates (the same rework twice is two events), .lock files not copied. Dry run showed 211 test records, 64 serials, 64 label records, 40 bag labels, 13 rework actions, and one serial "conflict" on 344C39463335 that is benign - assigned 09260018 locally yesterday and re-issued 09260018 by the server today. NOT YET APPLIED: writing to the production data folder needs Angel's approval.
2026-09-19 ALL SIX bricked Cow Robot boards recovered: 203B377D3045, 2042377D3045, 2033377C3045, 2044377B3045, 2068377B3045, 2067377B3045. Every one showed identical damage (RDP 0xcd, BOR_EN 0x0, nBOOT_SEL 0x1) and every one came back byte-identical to known-good 206E377B3045 with Robot_4.5.0 verified. Zero boards scrapped. The last three were recovered by Angel running python -m universal_tester.production.recovery --firmware examples/firmware/Robot_4.5.0.bin himself. Six-for-six identical damage is conclusive that the interrupted option-byte write lands at a fixed point in a deterministic firmware sequence, which is what the first-boot fix (commit dabfa9d) prevents. STILL OPEN: the BOOT0 pull-down. The firmware reads the BOOT0 pin and takes its BOOT_LOADER backdoor when it reads high, which is the best explanation for the ORIGINAL intermittent "CDC did not enumerate" failures and is a separate fault from the bricking. Until that is fixed, a programmer cannot be left wired to the fixture and every recovered board must have its SWD lead removed before retesting.
2026-09-19 Recovery tally, Cow Robot boards bricked by the first-boot option-byte corruption. FIVE of six recovered: 203B377D3045, 2042377D3045, 2033377C3045, 2044377B3045, 2068377B3045 (the last two by Angel running the CLI himself). ONE left: 2067377B3045. Every one showed the identical damage - RDP 0xcd, BOR_EN 0x0, nBOOT_SEL 0x1 - and every one came back byte-identical to known-good 206E377B3045. The identical-damage pattern across six independent boards is the strongest evidence that the interrupted option-byte write always lands at the same point, not random corruption.
2026-09-19 Recovery CLI fixed after Angel hit "No STM32 target on SWD" at 0.04 V: the board takes its 3.3 V from the tester's UUT1 and only the station menu was raising it, so the standalone tool now powers UUT1 itself for the duration (contextmanager uut1_powered, dropped in a finally) with --no-power for a board energised another way. Also: python -m universal_tester.production.recovery warned about the module already being in sys.modules because production/init.py eagerly imported it - the station now does from universal_tester.production import recovery as production_recovery and the eager import is gone. And with no terminal the confirm prompt raised a raw EOFError traceback; it now refuses cleanly and names --yes, because the erase is irreversible and must never happen by accident. Correct running total: FOUR recovered (203B377D3045, 2042377D3045, 2033377C3045, 2044377B3045), TWO left (2067377B3045, 2068377B3045) - an earlier message of mine said two remained when it was three.
2026-09-19 Board recovery is now a station menu option (5, RECOVER A BRICKED BOARD) backed by universal_tester/production/recovery.py, which also runs standalone as python -m universal_tester.production.recovery --firmware <path>. It reads the option bytes over SWD, names every field differing from known-good, states that recovery mass erases, and needs an explicit operator yes; it powers UUT1 first because the probe can only see a powered board, and powers down in a finally. Recovery only counts as success if the board ends up matching known-good - a flash that does not verify, or option bytes still wrong after, is an error. examples/recover-robot-board.sh was deleted, superseded. Recovered so far: 203B377D3045, 2042377D3045, 2033377C3045 (the last one end-to-end through the module). Two left: 2067377B3045, 2068377B3045. WORKFLOW LIMIT worth knowing: recovery cannot be automatic during a test run, because the ST-Link has to be wired to the board and an attached SWD lead holds BOOT0 high (PA14 is shared with SWCLK on STM32G0), so the board would fail the test anyway. It is a deliberate bench step: board fails -> attach probe -> menu 5 -> REMOVE the SWD lead -> retest. That only changes if the firmware stops reading the BOOT0 pin.
2026-09-19 Two more API-misuse bugs caught by TESTS THAT EXECUTE THE CODE PATH rather than its pieces: board_recovery_menu called UniversalTester.disconnect(), which does not exist (it is close()), and unpacked load_approved_firmware() as a 3-tuple when it returns an ApprovedFirmware dataclass with .path and .version_string. Both passed py_compile. That is now four runtime errors in one session that compiled and would have passed a green suite. RULE: when writing a new function that calls existing APIs, grep each callee's definition before trusting the call, and write at least one test that actually runs the whole function.
2026-09-19 Cow Robot BOOT0 finding, answers Angel's pull-up question. RESET/NRST already has a permanent internal pull-up on every STM32, so it needs nothing. BOOT0 CANNOT be fixed with an internal pull at reset: the hardware samples BOOT0 before any instruction executes, so a firmware pull-down is far too late. BUT the application ALSO reads the BOOT0 pin itself and jumps to its BOOT_LOADER backdoor - proved on boards 203B377D3045 and 2042377D3045 by setting nBOOT_SEL=1 nBOOT0=0, which forces the ROM to boot main flash, and watching DFU still appear 0.4 s after power-up with the firmware verified byte-for-byte in flash. That app-level read happens after boot, where an internal pull-down DOES work, so the firmware fix Angel asked about is real: configure PA14 with GPIO_PULLDOWN and a short settle before sampling it (and sample more than once). Likely the true root cause of the intermittent "CDC did not enumerate" failures: a floating or weakly pulled BOOT0 reads high at random, the app jumps to the bootloader, and the station sees no application CDC. An attached SWD lead makes it deterministic because BOOT0 shares pin PA14 with SWCLK on STM32G0. Option-byte alternative: nBOOT_SEL=1 + nBOOT0=0 makes the ROM ignore the pin entirely, and the station could enter DFU with the BOOT_LOADER CDC command instead of the BOOT servo - but that alone is NOT enough while the firmware still reads the pin.
2026-09-19 Second bricked board recovered: 2042377D3045, identical corruption to 203B377D3045 (RDP 0xCD, BOR_EN 0, nBOOT_SEL 1, PCROP_RDP 1). The corruption being IDENTICAL across boards, not random, says the interrupted option-byte write always lands at the same point - consistent with a fixed station delay cutting power at the same place in a deterministic firmware sequence. Its UID at 0x1FFF7590 decodes to the DFU serial as (UID0+UID2) then UID1>>16: 0x00120030 + 0x2030374D = 0x2042377D, then 0x3045, giving 2042377D3045. Three left: 2033377C3045, 2067377B3045, 2068377B3045. Station now reports a board sitting in DFU distinctly instead of "did not enumerate" (commit on sandbox, tests 409 pass).
2026-09-19 Board 203B377D3045 RECOVERED over SWD, procedure scripted at host-api/examples/recover-robot-board.sh (commit 7b30978). Steps: -ob RDP=0xAA (this MASS ERASES, it is the only way back from Level 1), then -ob nBOOT_SEL=0 nBOOT0=1 nBOOT1=1 BOR_EN=1 BORF_LEV=3 BORR_LEV=3, then -w Robot_4.5.0.bin 0x08000000 -v -rst. Its option bytes now match known-good 206E377B3045 exactly and the firmware verifies byte-for-byte. SWD flashing is 2.4 s, even faster than CubeProgrammer over DFU at 5.0 s. Five boards still to do: 2033377C3045, 2067377B3045, 2068377B3045, 2044377B3045, 2042377D3045.
2026-09-19 TWO GOTCHAS that cost real time on the Cow Robot bench. (1) The ST-Link must be plugged into a NORMAL USB port. On the Universal Tester's gated UUT2 output it enumerates (0x0483:0x3754) but macOS publishes ZERO IOUSBHostInterface children, libusb says "no interface found; setting configuration: 1" then "could not set configuration", and STM32_Programmer_CLI gives DEV_CONNECT_ERR - same inside and outside the sandbox, so it is not permissions. Moving it to a normal hub port fixed it instantly. (2) On STM32G0 BOOT0 SHARES PIN PA14 WITH SWCLK, so an attached ST-Link holds BOOT0 high; the recovered board kept coming up in DFU with the probe connected even after boot selection was forced to main flash (nBOOT_SEL=1 nBOOT0=0), which means the APPLICATION is also taking its own bootloader backdoor on a high BOOT0. Always unplug the SWD lead before retesting a board.
2026-09-19 The Cow Robot firmware's bootloader backdoor (Angel mentioned it; confirmed by strings). Robot_4.5.0.bin exposes a CDC command table containing RESET, IDENTIFY, BOOT_LOADER, REBOOT, SONAR_DEBUG, SONAR_RESET, STOP_MODE, plus config setters like REQUIRED_GOOD_FRAMES, SENSOR_MIN_VALID_FT, SENSOR_MAX_VALID_FT, NRF_AUTO_REPLY, ROPING_COW_SW. BOOT_LOADER jumps to system memory - the image contains 0x1FFF0000 and 0x1FFF0004, the SP/PC vector pair of the STM32 ROM bootloader. There is NO DBGMCU reference in the image, so it does not detect a debugger. USE: the station could enter DFU by sending BOOT_LOADER over the existing CDC instead of driving the BOOT and RESET servos, which would be faster and remove the fixture's tightest mechanical margin (boot press 1890 vs release 1800 us). Also present and worth knowing: STOP_MODE=%hu, an STM32 low-power mode that would drop current and kill USB - a plausible confound when reading a low idle current.
2026-09-19 CONFIRMED OVER SWD, root cause of the Cow Robot post-flash deaths, and it is FIXED (commit dabfa9d on sandbox). Read from dead board vs the known-good 206E377B3045: RDP 0xCD (Level 1) vs 0xAA, nBOOT_SEL 0x1 (BOOT0 from option bit) vs 0x0 (from PIN), BOR_EN 0x0 vs 0x1. RDP 0xCD is the giveaway - nobody sets 0xCD, anything that is not 0xAA or 0xCC reads as Level 1, so that word is GARBAGE: three fields corrupted in one FLASH_OPTR write, the signature of a write interrupted by power loss. Level 1 blocks the ROM bootloader (no DFU) and nBOOT_SEL=1 makes the BOOT0 pin inert (servo gesture useless); flash reads back "failed to read the requested memory content". The mechanism: flash_robot_firmware detaches the ROM bootloader, which resets the MCU and starts the application's FIRST boot (option bytes + EEPROM defaults), and reboot_robot_after_flash then cut UUT1 power within milliseconds, mid-write. Fix = the Cow Remote pattern: wait for the first boot's own CDC, hold a 2 s guard (enumerating does not prove the write retired), then cold-cycle 5 s and test the clean second boot; a board that never boots waits out a 5 s grace instead of being power-cut, because a board failing the check must be reported, not also bricked. Costs ~7 s per board against the ~46 s freed earlier. tests/test_cow_post_flash.py had ENCODED THE DANGEROUS ORDERING AS A REQUIREMENT ("test_cold_reboot_precedes_cdc_identification") - it now asserts the opposite invariant, and both new tests were confirmed to fail against the pre-fix revision. RECOVERY of the six bricked boards is STM32_Programmer_CLI -c port=SWD mode=UR -ob RDP=0xAA, which mass-erases and returns to Level 0, then restore nBOOT_SEL=0 and BOR_EN=1 and reflash; not yet run, it needs explicit approval because the mass erase is irreversible. ST-Link works only when plugged into a normal USB port, NOT the tester's gated UUT2 output.
2026-09-19 ROOT CAUSE (strong, not yet SWD-confirmed) of Cow Robot boards dying after a successful flash. STOP FLASHING ON THIS STATION UNTIL FIXED - six boards were killed on 2026-09-19: 2033377C3045, 2067377B3045, 2068377B3045, 2044377B3045, 2042377D3045, 203B377D3045. Evidence chain. (1) The DMM shows the board is HEALTHY going in and dead coming out: 203B377D3045 read 15.453 mA at the initial current check, flashed and verified, then failed CDC; its next two runs read 1.883 mA and could not even reach DFU. Same signature on 2067377B3045: 15.160 mA, flashed, CDC failed, then 1.937 mA twice. A running idle app draws 12.4 mA, so ~1.9 mA means the core is not executing. (2) Both firmware images contain the STM32 OPTION BYTE unlock keys FLASH_OPTKEYR 0x08192A3B and 0x4C5D6E7F, plus FLASH_OPTR 0x40022020 - three times in Robot_4.4.0 and NINE times in Robot_4.5.0 - and PWR_CR1 0x40007000 six times. So the application writes option bytes and uses low-power modes. (3) The Cow Remote station already documents this exact behaviour in reboot_remote_after_flash: it waits for the app's first boot to COMPLETE (wait_for_remote_cdc) and only then powers off, with the comment "The application has completed its first boot (option bytes, EEPROM defaults)". (4) The Cow Robot station does the OPPOSITE: flash_robot_firmware detaches the ROM bootloader, which resets the MCU and starts the application's first boot, and reboot_robot_after_flash then cuts set_uut_power(1, False) within milliseconds. Yanking power while the app is mid option-byte write is exactly how nBOOT_SEL / BOOT_LOCK end up wrong, and either of those makes DFU unreachable over USB forever. That matches the symptom set precisely: good rail, ~1.9 mA, no CDC, no DFU, unrecoverable. PROPOSED FIX: port the Cow Remote first-boot pattern to the robot - after flashing let the application finish its first boot undisturbed, wait for its CDC, and only then cold-cycle and test the clean second boot. NOT a regression from the 2026-09-19 speed work: 2067377B3045 died at 20:50 on the old code; POST_FLASH_POWER_OFF_SECONDS and POST_FLASH_BOOT_SETTLE_SECONDS were never touched. Note Robot_4.5.0 was pinned only on 2026-09-19 (replacing 4.4.0, which lacks LED_CURRENT_MODE) and it has 3x the FLASH_OPTR references, which is a plausible reason this started biting today.
2026-09-19 STLINK-V3 on this bench will not connect: it is plugged into the Universal Tester's gated UUT2 USB output, and although it enumerates (VID 0x0483 PID 0x3754, serial 003F00253532511131333430, visible only after set_uut_power(2, True)) macOS publishes ZERO IOUSBHostInterface children for it, so libusb reports "no interface found; setting configuration: 1" then "could not set configuration" and STM32_Programmer_CLI gives DEV_CONNECT_ERR. Same failure inside and outside the Claude sandbox, so it is not a permissions problem - the device is not being configured, which points at the tester's switched VBUS not carrying an STLINK-V3 properly. Fix to try: plug the ST-Link into a normal USB port (the Anker hub or the Mac) and leave only its SWD wires on the board; it does not need to hang off the tester's switched output.
2026-09-19 Cow Remote station speed work. The CubeProgrammer flash path now lives in the SHARED universal_tester/production/dfu.py (flash_stm32_dfu_image), which both stations already call, so Cow Remote gets the ~11 s saving for free and cow-board-test.py's private copy from a218cd5 is deleted - one flashing path, not two. The shared version keeps detach_stm32_dfu after a CubeProgrammer flash (the Robot wrapper had been skipping it because it cold-power-cycles right after). A CubeProgrammer that is present but FAILS raises and is never retried with dfu-util, because that would turn a real flash failure into a slow pass. Also cached the remote's pyvisa lookup. KEY FINDING: the Cow Remote station was already written carefully and has almost nothing else to win - no flat sleep at or over 0.5 s anywhere in the file, no 20-sample rail averaging, LED_USB_OPEN_SETTLE_SECONDS already 0.03, USB_ENUMERATION_POLL_SECONDS already 0.02, LED_CURRENT_SETTLE_SECONDS already 0.05. The Cow Robot station was the outlier, not the norm. Do NOT touch POST_FLASH_FIRST_BOOT_OFF_SECONDS = 5.0; that is the deliberate first-boot option-bytes/EEPROM handling from 2026-09-16. Remote baseline for reference: 77.6 s median over 107 passing runs in examples/log-cow-remote. Commit daef840 on sandbox, pushed. NOT hardware-validated: the robot board was removed from the fixture before the refactor could be re-run end to end, and no Cow Remote board has been on the bench at all - both need one confirming run.
2026-09-19 Cow Robot station BUG I shipped and fixed, commit bcb8736 on sandbox. Making the LED camera validation once-per-launch (a156c42) skipped the camera open from board 2 onward but left camera_format as None, while the REQUIRED HARDWARE DETECTED banner immediately after still did camera_format[0]. Board 1 passed; pressing "test next board" died with TypeError: 'NoneType' object is not subscriptable right after printing the PSU/DMM/Tester/DFU lines. Fix: cache the validated format itself, not just the device id, and drop the sentinel exception. Two lessons worth keeping. First, when you make something once-per-launch, grep every later use of the variable it used to set, not just the block you edited - this is the second time in one session that a patch compiled and passed 394 tests while being a runtime NameError/TypeError on the happy path (the other was the VISA cache write landing inside find_universal_tester_port). Second, the suite had no test that called discover_station_hardware more than once, so nothing could have caught a second-board bug; tests/test_station_discovery_reuse.py now calls it repeatedly and was confirmed to FAIL against the broken revision before being kept. Always verify a regression test against the bug it claims to cover.
2026-09-19 Cow Robot speed work, round 2, all measured on board 206E377B3045. Commits f65a528 and a156c42 on sandbox (with 6209e59 and a218cd5). LED current loop 15.6 s -> 6.12 s: the 0.35 s post-CH1 settle was ~35x too long (current is stable within ~10 ms of CH1 enable, profiled with the DMM free-running) so it is now 0.10 s; the 0.75 s USB-off window was profiled working at 0.30 s and is now 0.40 s; and the flat 1.0 s before each re-identify is gone because open_and_identify_robot already polls. Validated against production history rather than by eye - two reps plus a full integration run, every state inside the band logged by passing boards, increments red +1.40 / orange +1.40 / green +0.80 against history 1.41 / 1.41 / 0.81. Also cached the pyvisa resource lookup (list_resources is a measured 1.02 s and ran twice per board for two bolted-down instruments; now 1.14 s on the first board and 0.000 s after) and made the LED camera validation probe run once per launch instead of per board, since the LED test repeats the identical format check for every board anyway. GOTCHA that nearly shipped: a careless patch put the cache write inside find_universal_tester_port, where vendor_id/product_id do not exist - py_compile passes, the test suite passes, and it would have been a runtime NameError on the first board. Always grep the patched symbol's line numbers and read the surrounding function, not just the compile result.
2026-09-19 MEASURED AND REJECTED for the Cow Robot station, do not 'optimise' these again. UniversalTester(startup_settle_seconds=1.0) looks like a free second per board but it is load-bearing: at 0.2 s *IDN? returns 'EmbdedErs,V1.0' and at 0.0 s 'EdderanirsTester0200V1.0' instead of 'EmbeddedEra,UniversalTester-002,0001,V1.1.0'. The CDC stream is corrupt, not merely slow, and there is no retry around it. dfu-util -t 2048 is impossible (the STM32G0B1 ROM reports wTransferSize 1024). The dfu-util readback verify is not a cost at 0.53 s. Camera timing cannot be measured from outside the station: macOS TCC denies capture to both the Claude shell and the Claude terminal panel ('OpenCV: not authorized to capture video'), only the app that owns the permission can open it, so anything camera-timed has to be instrumented inside the station process. Remaining untouched and unmeasured: 4 x 0.5 s camera settle_seconds per board, the 20 flush frames, 40 capture frames, per-board printer connects, label printing and the server publish.
2026-09-19 Cow Robot flashing is 3x faster with STM32CubeProgrammer instead of dfu-util, over the SAME USB DFU connection - no programmer hardware needed. Measured on board 206E377B3045: dfu-util download 15.4 s + readback 0.53 s = 15.9 s, against STM32_Programmer_CLI write + full -v verify = 5.0 s. A dfu-util upload of the CubeProgrammer result was byte-identical to the source, so the flash proof is not weakened; the station now also requires the literal "Download verified successfully" line rather than trusting the exit code. Target one board with -c port=USB1 sn=<serial>; -l usb lists devices and prints a device count, which suits the station's refuse-if-more-than-one rule. CLI lives at /opt/ST/STM32CubeCLT_/STM32CubeProgrammer/bin/STM32_Programmer_CLI - glob it, never pin the version, or an ST update silently drops back to slow dfu-util. dfu-util stays as fallback and is still used for DFU discovery. End-to-end on hardware, three consecutive DFU-entry + flash + verify + cold-reboot + IDENTIFY cycles took 10.2 s each; that path measured 32.9 s before. Also cut LED_CURRENT_SAMPLE_INTERVAL_SECONDS 0.20 -> 0.05 (SDM3055 at NPLC 1.0 integrates 16.7 ms and returns in a measured 27.5 ms, so a 5-sample block drops from 1.00 s to 0.30 s, count unchanged) - measured at the noise floor with CH1 off, so verify led_current__average_a against history on the first real run. Commits 6209e59 and a218cd5 on sandbox, NOT merged, never run through a full station cycle. Estimated run now ~46 s against the 91.9 s baseline.
2026-09-19 CORRECTION to the earlier option-byte note: STM32CubeProgrammer DOES read and write option bytes over plain USB DFU, so SWD is not needed to inspect a board that can still reach DFU (dfu-util cannot - it exposes only "@Internal Flash" on alts 0/1/2 and 0x1FFF7800 reads "not readable"). STM32_Programmer_CLI -c port=USB1 -ob displ. The Cow Robot is an STM32G0B1 (device ID 0x0467, 512 KB, Cortex-M0+). Known-good option bytes from board 206E377B3045: RDP 0xAA, nBOOT_SEL 0x0 (BOOT0 taken from the PIN, legacy mode), nBOOT0 0x1, nBOOT1 0x1, BOOT_LOCK 0x0, NRST_MODE 0x3. The two that brick DFU entry are nBOOT_SEL=1 (BOOT0 pin ignored, so the fixture's BOOT servo does nothing) and BOOT_LOCK=1 (always boots main flash). Either one is unrecoverable over USB because you can no longer reach DFU to fix it - THAT case still needs SWD. Worth adding an -ob displ check to the station to catch a board whose option bytes are drifting before it ships. Camera timing could not be measured from a shell-spawned process: macOS TCC denies camera access to anything but the station app itself, so USBCamera open fails with "Unable to open camera device index 0" - that measurement has to run inside the station process.
2026-09-19 Cow Robot speed work, all measured on the bench with virgin board 206E377B3045. Baseline passing run 91.9 s median; steps sum to 70.9 s so ~21 s sits outside the measured brackets. Two fixed waits dominate and were cut on the sandbox branch (commit 6209e59, NOT merged to main, not run through a full station cycle yet): (1) ADC2 rail sampling was ADC2_SAMPLE_COUNT=20 x ADC2_SAMPLE_INTERVAL_SECONDS=0.25 = 4.84 s and runs FIVE times on the happy path; the rail is DC and 20x0.25 s reads 3.3466 V against 8x0.02 s at 3.3448 V, 1.8 mV apart in a +/-250 mV window, so 8x0.02 = 0.163 s saves 23.4 s; (2) open_and_identify_robot slept a flat 1.0 s before its first IDENTIFY and runs EIGHT times per board, but over six cold boots the app answers IDENTIFY a median 10 ms after the CDC port appears, so polling to the existing deadline saves ~8 s and is more forgiving of a slow board. Scan interval 0.5 s -> 0.05 s because enumeration lands at ~0.54 s. Total ~31 s off 92 s. Tests 394 pass / 1 pre-existing unrelated failure (test_server_client.py, also fails on main). Measured and REJECTED as dead ends: dfu-util -t 2048 is impossible because the STM32 ROM reports wTransferSize 1024; the readback verify is NOT a cost at 0.53 s (the DFU write is 15.4 s of erase+program and is hardware-bound). Still on the table, not done: LED step has ~11.6 s of per-state sleeps, UART_MUX_DWELL_SECONDS 1.5 s x 3 = 4.5 s, the camera is opened TWICE per board (a redundant validation probe plus the LED-test open), and all instruments/printers are opened and closed per board though they are station-fixed. Throughput context: operator gap between runs is a median 54 s (best 7 s), so effective cycle is 146 s = 24.6 boards/hour and no test optimisation alone beats ~66/hour.
2026-09-19 Cow Robot DFU/option-byte finding. A corrupted APPLICATION image can never block DFU: the bootloader is mask-ROM in system memory and runs before the app, which is why a board that fails DFU is not a firmware problem. But this ROM exposes only "@Internal Flash /0x08000000/256*02Kg" on DFU alt 0, 1 and 2 - there is NO option-byte interface - and 0x1FFF7800 reads back "Page is not readable". So if nSWBOOT0/nBOOT0/BOOT_LOCK ever get written wrong, the BOOT0 pin stops selecting the bootloader and DFU is unreachable forever over USB; only SWD can recover it. That is the one corruption mode that matches a board with a good 3.3 V rail and zero USB in either mode, and it is consistent with the Cow Remote note that this firmware family writes option bytes on first boot. Next step when a dead board is to hand: put an ST-Link on it, read the option bytes, and compare against a known-good board - the comparison needs SWD on BOTH because DFU cannot read them. Flash geometry for reference: 512 KB, 256 pages x 2 KB, base 0x08000000. A virgin board auto-enters DFU at power-up with no BOOT gesture, and after flashing 4.5.0 it enumerated as "The Real Cow" 6 ms after the settle.
2026-09-19 Cow Robot "flashes then CDC fails" investigated on the bench. NOT a firmware regression: Robot_4.5.0 passed 21 of 37 runs today. 10 of 37 died on USB enumeration - 5 at DFU (pre-flash), 4 at app CDC (post-flash), 1 tester-side errno 6 "Device not configured" - and the failures clustered late in the session. The board left on the fixture presents NO USB device at all: no app CDC and no DFU, across 8 cold UUT1 power cycles (2 s and 3 s off) and 21 BOOT/RESET servo pose combinations (boot 1750-2050 us x reset 1150/1100/1050). VAR_OUT1 reads 3.353 V throughout and drops to 0.39 V when UUT1 power is cut. Ruled out: app firmware (DFU is ROM bootloader, independent of flash contents), servo pose calibration (swept), post-flash power-off timing, and a stuck RESET arm (servos powered and parked at reset=2000 us, boot=1000 us). Remaining suspect is the USB D+/D- path to the DUT - fixture pogo contacts, the USB passthrough, or the tester's UUT1 data gate - since power pogo pins can seat while the data pair does not, which explains a good rail with zero enumeration in either mode. Needs a physical check: reseat the board, inspect/clean the D+/D- pogos, and try a known-good board plus a reseated tester USB cable. Gotchas found while reading the code: the tester and the Cow Robot share VID:PID 0483:5740 and are told apart ONLY by the USB product string ("The Real Cow" vs "UniversalTester-002"), so a board that enumerates with a missing/changed product string would look identical to "did not enumerate"; and the servo press/release margins are very tight (boot press 1890 vs release 1800 us = 90 us; reset press 1150 vs release 1275 us = 125 us). Bench left safe: servos parked, CH2 off, UUT1 power and data off. Scripts used are in the session scratchpad, not committed.
2026-09-19 Cow Robot station LED camera 'light' gate: known-good board 206B377B3045 reads target bright fraction red 0.93 / orange 0.89 / green 0.87 against the 0.90 limit calibrated 2026-08-08 (August boards read ~1.00). Identical numbers on firmware 4.8.4 and 4.5.0, microscope LEDs on or off, and at the best possible ROI placement, so it is not firmware, the microscope light or alignment; the scene is simply brighter/differently framed than in August (whole-frame OFF V p90 102 -> 144) and the cause is unknown. Re-tuned min_target_bright_fraction to 0.847 in host-api/examples/robot-led-camera-settings.json with the repo's own formula (optical_color_tuning: max(nontarget_ceiling+0.05, active_floor-0.02); floor 0.867, ceiling 0.720). A dead LED still fails (fraction ~0.63, delta ~0). Gotcha: production-firmware.json (added 2026-08-26) pinned Robot_4.4.0.bin, which lacks LED_CURRENT_MODE, so the LED test could not run; now pinned to Robot_4.5.0.bin, the build all August passes used. Station also selects the camera by name (VMS800S) because Continuity Camera shifts OpenCV indexes.
2026-09-16 Cow Remote station: "0 neutral V1 packets" after a flash only ever hit a board's first-ever boot (option bytes + EEPROM defaults written on that boot); operator restart cleared it. Fix in host-api/examples/remote-board-test.py: reboot_remote_after_flash now lets the app boot once, powers USB1 fully off for 5 s, and tests the second boot. Not yet verified on hardware.
2026-10-01 UT-002 rev A prepared for release (not tagged): ERP PCB EE-P-0006-A, assembly EE-A-0009-A (pk 243), doc EE-D-0001 (pk 244), 40 BOM lines loaded with erp-bom-import; new parts EE-C-0198..0214; schematic retargeted to EmbeddedEra library symbols carrying IPNs on branch ut-002-release-a; ERC/DRC no worse than before (2 pre-existing LM66100 OR-ing ERC errors). SOP-008 2.1, SOP-002 2.3. Tag hw/universal-tester-002/A waits for second-person review (SOP-021 step 11).
2026-11-07 Server side: tester-api's 120 s retry loop stopped advancing after one board (no journal output; process up since 11-02); drained by stopping the service and running process_pending_events by hand, then restarted. Root blocker was ERP 33R stock (see mrp). Release/scrap dispositions for 10260007/079/016 written as plex-admin quarantine events at owner's instruction (no retest). Owner wants the quarantine/rework receipt print to be optional.