plex-server¶

The home server itself: Ubuntu 24.04 on a Dell OptiPlex 7050, hostname Plex, 192.168.5.15. SnapRAID (3 data + 1 parity) under mergerfs, ~36 docker containers, Immich, Plex/Jellyfin, the Garmin→Calendar scheduler. Repo: not a repo — the working directory is /home/angel itself.

The notes for this machine are not in this vault. They live in an ICM system map at ~/Notes/map/ (inside this vault, same repo), because they document a live machine rather than a codebase: object cards for the nouns, process cards for the verbs, and a change-impact index.

Status¶

2026-10-31: Owner requested smarter listener alerts. sec-watch now verifies UDP owner executable + UID + socket inode against narrow Chrome temporary-port rule; unknown/TCP still alert. Removed 6 Chrome numeric baseline entries; baseline refresh excludes verified temporary sockets. Cron saves last 48 private evidence snapshots with socket/cgroup/owner data before notification. Six offline tests + real wildcard Python socket detection/evidence-retention check passed; full --check clean. Details in security-watch. No helper model ran (enforced delegation scope unavailable).
2026-10-31: SOP-049 listener alert triaged from live socket cgroups + Home Assistant socket/process mapping. All ports attributed to existing HA/go2rtc, cloudflared, Avahi, NordVPN, and transient Chrome sockets. Refreshed stale listener baseline (48); sec-watch --check → ok: no findings. Evidence + old baseline: ~/.local/state/sec-watch/review-2026-10-31/. Random UDP port churn can recur; details in security-watch.
2026-09-08: storage hardening at v9 — 15 of 21 items done. Item 05 (off-box copy of photos + EmbeddedEra) is the only remaining permanent-loss risk. Items 15, 16B, 17 run unattended this week from ~/maintenance/.

Decisions¶

Version history is a restic repo on the pool, not rsnapshot's tree in parity: 3.5M hardlinks rotating hourly can never be parity-protected (45k-file array, 500-deletion precheck limit); a few thousand immutable pack files can. Same repo replicates off-box unchanged. 2026-09-08.
Server facts live in ~/Notes/map/ cards, one home per fact; this vault keeps user/project context and links to them. Memory under ~/.claude/projects/-home-angel/memory/ stays for user and feedback facts only.
Root work goes through a scoped, revocable /etc/sudoers.d grant the human installs, then revokes — never blanket NOPASSWD. See ~/Notes/map/processes/root-maintenance.md.
Long maintenance runs as root systemd one-shot timers writing markdown reports, not as agent sessions — zero tokens, survives a dropped SSH connection.

Gotchas¶

History protection is "any one drive, from 2026-09-09 forward": the restic repo enters parity at the first 03:30 sync after each snapshot; pre-2026-09-08 history (rsnapshot, back to March) is still disk1-only; same-day writes are unprotected until 02:20/02:50/03:30; the NVMe case needs the restic password in the password manager. map/objects/storage/history-repo has the table.
sudo needs a password and tool shells have no tty: sudo fails silently in agent calls, and interactive commands like gh auth login hang forever rather than erroring.
Claude Code's background-Bash memory supervisor kills long jobs here — free reads ~300 MB because qbittorrent and page cache hold ~11 GB reclaimable. Run long things in the foreground or from cron.
Never a bare docker compose up -d (the samba service has drifted from the live samba-nas container) and never a bare snapraid sync (the guard + precheck exist to stop a mass delete reaching parity).
Read SMART history from /var/lib/smartmontools/attrlog.*.csv — world-readable, no root, and it does not wake a sleeping disk.
Ponytail (v4.9.0): /plugin commands must be typed at the Claude prompt; typed into bash they fail with No such file or directory. Its hooks are node-based and only load on a fresh session.
A user oneshot unit that starts a self-daemonizing process needs KillMode=process, or systemd kills the child when the oneshot exits (bit codex-remote-control 2026-10-11).

Open¶

See ~/Notes/tasks.md — this section is now generated from the task register.

Log¶

2026-11-11: NAS account yara created, read/write on all of EmbeddedEra (owner: "show everything to her").
2026-11-10: Completed read-only scheduled-job storage audit and proposed fix plan: server-cron-space-audit-2026-11-10, per-job inventory server-schedule-storage-inventory-2026-11-10. No production policy or schedule changed. Main findings: archive copied back into precious, usable-capacity forecast mismatch, failed Docker snapshot accepted, obsolete USB root cron, overlapping backup histories. Private evidence under ~/.local/state/server-audits/cron-space-audit-2026-11-10/.
2026-11-09: assistant PR #98 merged (548b4e3), radar retries openrouter/free 3x. Verified with a real scan at 17:20: 231 scored, 1 contact, rc=0.
2026-11-09: assistant PR #97 merged (3bd2905): every pod vault page takes .git/notes-sync.lock; radar moved to the openrouter/free router.
2026-11-09: Dropped five old vault autostashes after confirming their hand-written lines are in the vault; patches in .git/rescue-20261109/stashes/.
2026-11-09: cockpit-index cron now runs under flock -o -w 120 -E 0 on .git/notes-sync.lock (skips when busy); pod.activity journal pages take the same lock (30 s wait, then skip; regenerated from ledger). Prior crontab: ~/.local/state/crontab.before-vault-lock.
2026-11-09: Vault stuck mid-rebase since the 12:00 notebook-publish sync (tracked files written during the pick). Recovered with git rebase --quit + checkout -B master, dropping local generated-only commit d0a5fb06; uncommitted edits kept and pushed as b9fccdfd; 14:40 publish succeeded. Patch backup in .git/rescue-20261109/.
2026-10-27 12:31: publish-note refusals before anything is written (dirty tree, origin fetch, guard, failed commit) exit 3 → retry/swap instead of uncertain; existing prod file stays exit 1 (assistant c58f938, suite 3030 passed). SOP-018 v7.4.
2026-10-27 12:19: precious-sync.sh with displace_type_flips installed by owner; verified identical to ~/rescue-kit, syntax ok. Handoff (alerts chat closed, nothing in flight): all three alerts fixed and proven (precious run green, engineering deploy complete, pt100 note live HTTP 200, content-reconcile all ok). Optional leftovers, not asked for: delete branch rescue/pt100-2026-10-27 in ~/embeddedera-site and ~/.local/state/rescued-og/ after a week; content-release alert key includes the reason (duplicate pages); publish-note's "deploy failed after the build" path leaves the OG png and needs manual reconcile. First real test of retry/swap is the next failed morning release.
2026-10-27 10:02: pt100 note posted late on owner request (link to the 10-28 note made plain text; owner_amend_today + late-release). Owner: "we can never miss a note" → chose retry until 08:30, then swap in the next ready note (assistant 4a59781); preflight now builds each day separately (~/.local/bin ba7ab30). SOP-018 v7.3.
2026-10-27: Fixed three alerts at the root.
precious-sync: a kicad-fork ref flipped dir→file → rsync code 23 every night. Fix: move type flips to precious/.displaced/<date>/; new script in ~/rescue-kit/precious-sync.sh awaits sudo install. Today's run is green.
engineering-deploy: an 81 MiB .kicad_pcb went over the Cloudflare Pages 25 MiB file cap. Fix: build skips files >25 MiB and links GitHub instead; deploy drops any that slip through. ~/.local/bin b2dc178.
Daily content release: the pt100 note linked the next day's note; the all-notes preflight build hid it. Fix: per-day link rule at approve, preflight and release (assistant 995757a); publish-note keeps the deploy log and restores the tree when the build fails; new note_queue reconcile-not-published (b7b19fa). pt100 is reconciled → missed, so the 10-28 prepare re-reviews it for 11-04. Failed commit kept on branch rescue/pt100-2026-10-27 in ~/embeddedera-site.
2026-10-26: Vaultwarden key push live — server@ account in org Plex Server, vault-push nightly 02:47 (61 key files as Secure Notes; create/update/skip proven; failure hook proven). Card: map/objects/containers/vaultwarden.md.
2026-11-02 19:05: NUT verified: nut-server/monitor/driver active, upsc cp1500@localhost OL 100% ~54 min, upsmon logged in, ntfy test push sent. Cards updated (machine, hardening 20, alerting). Unplug test passed 18:55 (on battery 20 s, back on line). Left: BIOS AC-recovery.
2026-11-02 18:55: UPS on battery outlet, USB seen (CyberPower CP1500 AVR, 0764:0501). Post-boot checks clean (6 mounts, serials match, all containers back, hostfw active); qbittorrent held by kill switch at boot, re-arms itself. NUT installed but MODE=none; owner to run sudo bash ~/maintenance/ups-2026-10/nut-setup.sh (standalone, ntfy on battery/low/shutdown, backup of /etc/nut). Then verify upsc cp1500@localhost, update machine (says "no UPS") + hardening item 20. Ubuntu 26.04 offered; audit says no 26.04 move before 2027.
2026-11-02 18:15: UPS install started. Pre-checks clean (snapraid sync/scrub inactive, no backup jobs, 72 containers; snapshots in ~/maintenance/ups-2026-10/*-20261102.txt). Agent ran shutdown -h +2; after boot run the checks in ~/disk-map-20261026.txt, then nut.
2026-10-26: Immich v2.4.1 → v3.3.5 (security release; CA compromise + Node patches). IMMICH_VERSION pinned to v3. Pre-upgrade dump + rollback image tags recorded in map card immich.md.
2026-10-26 12:45: UPS bought (audit item 6 / hardening 20).
Plan: a 15–20 min power-off at a time the owner picks when home. Suggested: 14:00 or Tue 10:15; avoid 01–05 and 18–23.
Before: optional Ubuntu updates incl. docker. The agent runs shutdown -h HH:MM itself (polkit ok, no sudo).
Owner, at the box: server on a battery outlet, UPS USB cable to the server.
After boot (agent): the checks in ~/disk-map-20261026.txt (containers list in ~/maintenance/ups-2026-10/). Then nut, which needs the owner's sudo apt install nut and the UPS model.
BIOS AC Recovery was already on.
2026-10-26 11:20: audit item 8 done without root.
Change: vaultwarden/data/ and ~/torrentvpn/nginx-proxy-manager/ 775 → 700, and .env.backup-20260116… 664 → 600. Undo is in audit-2026-10-26/perms-undo.txt.
Proof: a uid-65534 container is denied all 5 files. angel's sqlite3 .backup of the vault still works (quick_check ok, 1 user, 14 items), vault /alive and NPM both return 200, and vaultwarden is healthy. So the root 640 root:angel step is no longer needed.
Still open: the filebrowser login (uid 1000, mounts all of ~, API 401 without a token) reaches these files. ~ is still 755 (samba-nas/plex check first).
2026-10-26 11:30: full-audit handoff continued.
Step 1 verified. xrdp, Cockpit and PCP were disabled at 10:49, and 3389, 9090, 4330 and 44321-3 are closed. 50-no-x11.conf is in place. hostfw is active and enabled at boot (observe).
Step 2 not run. The Ubuntu updates are still not done: apt history shows only unattended-upgrade, last run 10-23. Owner.
First would-drop sample (37 lines). The log is rate-limited, so the real volume is unknown. Every line is inbound UDP from SPT 443/53 to an ephemeral port: QUIC from Meta and Anthropic, and DNS from 8.8.8.8. These are late replies after conntrack's UDP timeout (30/120 s). conntrack is at 1324 of 262144, with no "table full".
notebook-publish #338 came from 3 notes-sync runs racing at 11:00:41 ("Cannot rebase onto multiple branches"). notes-sync now re-runs itself under flock -o -w 120 on .git/notes-sync.lock. The -o means a detached auto-gc never inherits the lock (2578 of 6700 loose objects). It runs unlocked on macOS. Proof: 3 parallel runs gave 3 × exit 0; with the lock held 6 s it waited, then synced; a relative ./notes-sync -n works. notebook-publish now copies its log tail to stderr on failure (ops-scripts 7c3f2d4, pushed together with 4bc3310). A manual run ended success, and #338 auto-closed.
New snapraid-age (ops-scripts 6cdd8b3). It exits 1 when the last snapraid-sync run failed, or when no sync has finished OK for 36 h. Under cron-run that becomes one job-failure incident: a page and a Kanboard task. All 3 outcomes were run by hand, and job-failure classifies the failures as permanent/escalate.
snapraid-age is not live. Installing the crontab was blocked by the auto-mode classifier. The owner runs crontab ~/.local/bin/crontab.txt, then jobs-index.
2026-10-26 HANDOFF (full audit; chat closed at the size limit). Report: ~/maintenance/audit-2026-10-26/AUDIT.md (top-10 + 12 areas + rounds 2-3). Done + proven today: SnapRAID back in sync (10:21 Everything OK; excludes /claude/ee-meet-cache/, node_modules/); hub login (sandbox 6880bce promoted; 401/200); arr auth Enabled (via NPM 401/200); mosquitto auth (anon refused); erp-backup verify quick_check + vault >=1 user/item (~/.local/bin 4bc3310, PASS); Docker old build cache cleared. Root reports 1+2 read (no host firewall; sudoers.d clean; SUID//etc/OOM clean). Waiting on owner: (1) sudo bash ~/maintenance/audit-2026-10-26/fix-root.sh services observe (xrdp/Cockpit/PCP off, X11 off, firewall log-only); (2) Ubuntu updates: apt-get upgrade with docker held, check reboot-required, docker-ce/containerd separately in a window (restarts all containers; then verify docker ps + gluetun/qbittorrent), snap refresh; no 26.04 move before 2027; (3) rotate samba password + cloudflared token (both seen in ps), then move to 0600 files; (4) fix-user.sh tdarr (Tdarr unused) + is postiz retired; (5) save ~/.config/claude-hub/env + ~/.config/mosquitto/env in Vaultwarden; (6) Plex 32400 is publicly mapped: intended? UPnP on?; (7) UPS. Next agent:* after owner runs (1)/(2), verify ports closed + hostfw table loaded + all containers up; 11-02 read the would-drop log (Open item); then remaining top-10: image updates + replace dperson/samba, Vaultwarden DB 640 root:angel (NOT 600: erp-backup reads it as angel), Portainer docker.sock, healthchecks, swap/zram, DHCP reservation, fstab nofail, snapraid-alert opening a Kanboard incident + sync-age >36 h alert. Design item: angel in docker group = agent is root (rootless docker / socket proxy / separate user).
2026-10-26: Owner said "do 1": erp-config PO template edit committed + pushed (e68144e, "Send the tracking to purchasing@"). erp-config is source only; whether the live InvenTree PO report template has this text is unchecked (container read blocked by auto mode).
2026-10-26: Repo-sweep handoff continued. Done + proven: eeroute native-usb-pair-skew WIP committed (new unittests clean) and pushed (efba0f7, draft PR EmbeddedEra/eeroute#1); erp-config tester-quarantine-reports pushed (0e3d871); kitchen-hub local main fast-forwarded 40 (untracked server/nightowl-cameras.sh was identical to origin). Cow_Robot_Board: the 5 local branches (engineering/repos clone) predate the 10-06 history rewrite (no merge base) and main already has C15=10uF, J10 in_bom no, product.yml 004B, so nothing to merge; deleting them is optional. branch-cleanup-2026-10-26.sh NOT run yet (85/85 branches still on remotes, 3 autostashes). Owner calls left: erp-config uncommitted PO line ("invoice and tracking" -> "tracking", since 10-01); merge UT#12, Boost_Gauge#1, ee-meet#1 (all MERGEABLE); kicad-lib 23 unmerged branches (20 docs/native-*, erp-httplib, floorplan-scorer, sti2004-parts).
2026-10-26: firewall plan: owner says LAN = family only (guest Wi-Fi separate), xrdp/Cockpit unused, 'only 51820 forwarded', but Plex reports 32400 mapped publicly, so the allow-list is 51820/udp + 32400/tcp. fix-root.sh services + observe (log-only nft prerouting table, priority -150, never flush ruleset); enforce after a week of logs. Vault cert = Cloudflare DNS challenge (no inbound needed).
2026-10-26 10:25: SnapRAID back in sync (owner ran the reviewed sync 10:21, Everything OK, -101,573 cache files; excludes /claude/ee-meet-cache/ + node_modules/). Mosquitto auth live (anon refused, user ok). Root report 2 clean (SUID, /etc, OOM, ssh fails 0). New: Cockpit :9090 on LAN; angel in docker group = agent is effectively root. No global IPv6.
2026-10-26 10:05 owner ran fix-user.sh: hub login live (401/200 proven), *arr auth Enabled (via NPM 401 nokey/200 key); mosquitto section aborted (config dir uid 1883) → rewritten to write inside the container, rerun fix-user.sh mqtt. Root report (partial): no host firewall confirmed (no ufw, INPUT ACCEPT, DOCKER-USER empty); sudoers.d clean; Cockpit PCP pmcd/pmproxy on 0.0.0.0 (new). SnapRAID excludes added by owner; sync steps 2-4 pending. Follow-up collector root-collect-2.sh.
2026-10-26: audit fixes. Done + proven: erp-backup verify now quick_checks every restored SQLite and needs >=1 vault user/item (verify PASS 1/14; ~/.local/bin 4bc3310 — the audit's 'Vaultwarden never restored' was wrong, it was restored with min 0). Hub cookie login on sandbox branch sandbox/hub-auth-20261026 (6880bce, 15/15 checks on a throwaway instance). Docker: dangling/old build cache cleared (~1 GB); the '38.6 GB reclaimable' is tool images + stopped stacks, not waste. Live-service changes (hub deploy, *arr, mosquitto, tdarr retire) blocked by the auto-mode classifier → fix-user.sh for the owner. Do not chmod 600 the vaultwarden DB: erp-backup reads it as angel.
2026-10-26: full read-only audit (security, agent surface, data/DR, hardware, reliability, capacity, network, drift, governance) → ~/maintenance/audit-2026-10-26/AUDIT.md. Top find: parity stale since 10-23 (snapraid-sync failed 3 nights, ntfy fired, nobody acted — alert without a ticket). Current: 39 GiB RAM with 19 GiB available (the split to-do's 16 GB is its dated 09-24 measurement), 81 containers (72 up). Map drift 31 cards (was 26). Nothing changed on the server.
2026-10-25: sec-watch net.stratum false positive = HA androidtv integration (Insignia Fire TV 192.168.4.87:5555, ADB). Port-only miner matches now skip private IPs (BAD_IPS still checked). New listeners 8123/6054/18555 = Home Assistant + go2rtc.
2026-10-25 HANDOFF (storage plan, chat closed at the size limit). Done + proven: storage-forecast auto-cleanup (--test ok; all 6 rules ran clean under a bare cron env; first run freed 8.7 GB, / 442.7→434.0 GB = 86.5 %), growth attribution, exact R2 (6.4 GB), rescue stick self-managed (#337 closed), Monday storage line (assistant 4a9cbba, 3022 tests pass), precious-sync excludes EmbeddedEra/backups (~/rescue-kit copy ran clean: 40924 files, 0 missing, 0 mismatches), deploy guards (publish-note, therealcow, notebook/sop notes-sync first: guard refused once on a failed sync, then deployed). Commits ~/.local/bin c8232d0, assistant 4a9cbba (pushed). Memory: storage-budget-policy; SOP-043 v1.3 deployed. Next: (1) confirm the owner ran the two sudo lines (diff -q ~/rescue-kit/precious-sync.sh /usr/local/bin/precious-sync.sh; journalctl --disk-usage ≈1 GB); (2) check tomorrow's 08:00 storage-forecast output + 02:20 precious-sync log; (3) owner decisions on task #336 (old precious ERP sets ~11 GB, immich dumps 3.8 GB, kicad-fork checkouts 13 GB, story-android 25 GB); (4) 2026-11-01 switch on cleanup (touch ~/.config/storage-forecast/reclaim-enabled) unless the owner says sooner.
2026-10-25 #336 done + proven: owner ran the root script. / 402→347 GB used (79 %, 55 GB freed); journal 982 MB under the new 1 GB cap (journald.conf.d/size.conf); all target paths gone; precious/immich/backups 64→20 (14 = pool + 6 Aug only-copies); storage-forecast --test ok. Kanboard #336 closed. eeroute-status now errors on the fork section (no ~/kicad-fork) until re-cloned.
2026-10-25 Handoff step 1: precious-sync install verified (diff -q identical, root-owned 10:11). journald cap NOT applied — /etc/systemd/journald.conf.d/ empty, journal 4.0G; re-handed one-line sudo.
2026-10-25 #336 owner approved deleting the old precious ERP sets, immich dumps, kicad-fork checkouts and story-android ("backed up elsewhere"; story-android cancelled). Verified first: 25 precious-only erp sets + 40 precious-only immich dumps found in restic; 6 Aug 20-25 immich dumps (346 MB) NOT in restic -> kept. kicad-fork WIP committed to wip/waypoint-diagnosis-2026-10-20, all branches pushed to /mnt/storage/EmbeddedEra/Github/kicad-fork.git (verified by sha). /var/tmp/erp-restore-* (20 plaintext restore dirs, 5.6 GB) came from erp-sandbox up never cleaning up; fixed (~/.local/bin commit). One root script handed to the owner (journald cap folded in).
2026-10-25 Storage plan built (owner: "do everything"). storage-forecast: allow-listed auto-cleanup before any task (report-only until ~/.config/storage-forecast/reclaim-enabled), growth attribution, exact R2, rescue stick self-managed (#337 closed). precious-sync excludes EmbeddedEra/backups (needs ! sudo install). Monday job report gets a storage line. Deploy audit: therealcow guard added; notebook/sop deploys notes-sync first; home/engineering/weekly-research already safe. surfaces-check retries once. Manual cleanup freed 8.7 GB → / 434.0 GB (86.5 %), still over 85 %. Dropped docker daemon.json GC (needs a docker restart = all containers).
2026-10-25 09:00 offsite-backup green: pod art rewritten out of R2, account 9.15 → 6.37 GB (object listing). Guard now sums the object listing; R2 /usage lagged hours and kept refusing after the prune (~/.local/bin 72697b1). Incidents all closed.
2026-10-25 Owner decision: print-on-demand art (pod/designs, pod/snapshots) stays out of Cloudflare R2. Its copies: local restic history on the pool (verified 2321 files / 3.86 GB in the 02:50 snapshot) + Printify/Etsy. Excluded in offsite-backup (~/.local/bin). Site redeploy done by owner: main in sync, PR #9 live (cached slots 0.3 s). Cache rewrite done (latest snapshot 6.9→6.0 GiB).
2026-10-25 Morning alerts root-caused. (1) Daily note release deployed embeddedera-site from a local main 4 behind origin → rolled PR #9 (booking D1 cache) off the live site, push refused → ValueError. Fixed: publish-note fast-forwards to origin/main or refuses (assistant cf870a8, tests); local main rebased (ahead 2, unpushed); redeploy + push pending owner (auto-mode blocked prod deploy). (2) offsite-backup refused at 9 GB: pod art ~2.5 GB growing + ~0.9 GB caches the case-sensitive exclude missed. Excludes fixed (~/.local/bin 66eb90a); snapshot rewrite+prune pending owner (blocked as cloud delete); pod-art off-site policy = owner decision (task #335). (3) booking-warm/Kanboard false pages: frequent jobs defer any failure; heartbeat retries. (4) storage-forecast now opens Kanboard tasks.
2026-10-25: Homepage → v2.7.0; Home Assistant tile + custom neon icon (homeassistant.lan/ha.lan, NPM host 29); NAS pool space in resources widget (/mnt/storage ro); Samba NAS tile. HA trusted_proxies needs root install of staged config.
2026-10-23 15:30: legacy tasks-digest-alert retired (owner OK; owner ran the unit edit, classifier blocked it). 7 timers now OnFailure=job-failure@ only; tasks-digest-alert@ is the handler-crash fallback. 14:08–15:18 alert storm was my own Log line quoting the scanner-tripping phrase; reworded. Lesson: never quote a secret-scanner match verbatim in the vault.
2026-10-23: 07:07–08:27 phone got 9+ "unknown-user-job.service failed" pushes. Cause: notebook-publish refused to build because the secret scanner (notebook-build:50) matched a Cloudflare token-scopes line (token, colon, scope list) in projects/embeddedera-site.md (false positive). Reworded to "token scopes (Pages, D1, Access)"; publish ok 08:31, incident closed 08:40. Follow-ups under Open.
2026-10-23 13:41: owner approved the claude-remote fix and keeping claude-remote over claude-rc-server. claude-remote now Type=exec (no RemainAfterExit, ExecStop removed); claude-rc-server disabled+stopped; update-claude.sh checks is-active 20s after restart. Owner did NOT approve retiring the legacy alert handler or the notebook-publish stderr change (still Open). See claude-tooling.
2026-10-18: KiCad native planning/pair-transition/tuning milestone verified and backed up; measured real-board limits and next work live in kicad-autorouter. No server configuration changes.
2026-10-18: removed the dead root-cron wg-watchdog.sh (pinged non-existent host wg0, tried a masked wg-quick@wg0; 90 false log lines in 3 h). Verified: no "WG watchdog" journal lines after the 17:54 tick. wg-docker-watchdog.timer is the real watchdog and stays.
2026-10-18: sec-watch alerts for tcp 8093 (a Claude session's temporary LAN PNG preview server, http.server --bind 0.0.0.0, killed with fuser) and udp 49646 (ephemeral client socket, source not identified; Chrome launched 14:10:43). Both gone on check. Idea: bind preview servers to 192.168.5.15, or make sec-watch skip ephemeral-range UDP owned by client processes.
2026-10-15: NAS home for call/meeting recordings: EmbeddedEra/communications/ (inbox/, year folders, register.csv, README), procedure SOP-048. Sits under EmbeddedEra/, so parity + hourly rsnapshot + nightly NVMe copy cover it (NVMe has 84 GB free, so watch it if video recordings pile up).
2026-10-15: Added the twice-weekly brief meeting (Remote Control session + calendar event + noon/17:00 nags) so the owner is made to pick notes/replies/leads/builds; tariff note queued at top of note-backlog; USB impedance calculator delegated (branch tool-impedance). Found 7 note drafts waiting on owner approval.
2026-10-15: Added cloud Market brief routine (Mon+Thu, web-only, no connectors) + market-brief-pull cron → notebook projects/market-brief; weekly-research reads its note ideas. Found the opportunity radar silently broken (free-model scoring 0 since ~10-01, 1071 unscored, exit 0) — repair delegated.
2026-10-15: Plex was missing from the environment picker at claude.ai/code: claude-remote.service uses --remote-control --bg (one session, no environment). Added claude-rc-server.service (claude remote-control server mode, same-dir, under script for a tty). See claude-tooling.
2026-10-15: Keep-learning loop for every area (owner: "keep building that knowledge, my style … the same for many other things"): where-lessons-go.md routes each kind of lesson (style, procedure, trap, approved example, said-twice) per area to one home; examples/<area>/ for approved work; rules/CLAUDE.md "Keep learning, in every area"; notes-daily now routes corrections and likes by the table and flags repeated corrections (episodic-memory search) as candidates for a check tool.
2026-10-15: iPhone Obsidian crash-looped at launch after a pull. Vault pinned the Git plugin at 2.39.0 (tracked in .obsidian/plugins/obsidian-git/), which overwrote the store's 2.41.1 on clone: updated the tracked copy to 2.41.1 and set autoPullOnBoot false (pull on the 5-min interval instead, not during app start). Recovery on the phone: open Obsidian in airplane mode, then pull manually.
2026-10-15: Vault images and PDFs moved to Git LFS (cb17773, 116 files, 54 MB; .gitattributes). The iPhone's Obsidian Git crash-looped on the 55 MB clone; a depth-1 clone is now 9 MB and the phone sees pointer files instead of pictures. Server (git-lfs 3.8.0 in ~/.local/bin, git lfs install --local) and any machine with git-lfs get the real files at the same paths, so links, notebook-build and brand-kit are unchanged. Mac/Windows need git-lfs installed (SETUP.md) or they also see pointers. GitHub free LFS quota is ample for this.
2026-10-15: bin/todo-done KEYWORDS (vault, every machine; symlinked in ~/.local/bin on Plex): finds to-dos already finished (deleted lines in git history, moved ones excluded), their Log lines, and open ones. Rule in rules/CLAUDE.md: run it before adding a to-do or asking the owner for something; notes-daily also deletes new to-dos that repeat finished work.
2026-10-15: notes-daily replaces notes-weekly (owner: end-of-day tool to keep the ICM map and memory organized; commit directly, no PR). Weekly had skipped since 09-13 behind unmerged PR #1. Inputs: owner-message digest of the day's interactive sessions (~/.claude/routines/session-digest.py) + commits + vault-lint; Sonnet pass; cron 23:00. New shim ~/.local/bin/episodic-memory. Card: map/objects/host/claude-tooling.md.
2026-10-14: Claude memory is now shared across machines: ~/.claude/projects/-home-angel/memory is a symlink to ~/Notes/claude-memory/plex-server/ (old dir kept as memory.pre-vault-20261014), rules/CLAUDE.md imports its MEMORY.md on every machine and says to write new memories there; notes-nightly skips the old copy step. Mail/doc guard hooks (claude-mail-guard) are still server-only.
2026-09-27: Oddly Fitting Goods Etsy POD shop built (assistant PR #8) and scheduled; launch waits on Etsy API approval + owner "go". See etsy-shop, SOP-042.
2026-09-27: Story cron fixed (user-bus env missing under cron); 27 Sep Facebook Story posted 10:00. See ../map/objects/host/story-android.
2026-09-27: Opportunity radar live (assistant PR #6): public feeds -> free OpenRouter models -> Monday notebook page + ntfy top 5; role business contacts only, nothing sends email. See opportunity-radar.
2026-09-27: Unattended daily content release live (assistant 413629a); Postiz scheduler worker outage found and fixed (orphaned orchestrator after reboots); watchdog now checks Temporal pollers. Details in content-pipeline.
2026-09-26: Android emulators now resume from a quick-boot snapshot: Play boot 24 s -> 9 s, avg CPU 86% -> 29%, ~10 s save on exit, 3.8 GB on disk. Login profile untested with owner. See ../map/objects/host/story-android.
2026-09-26: Emulator GPU A/B: hardware (-gpu host, Intel HD 630) boots in 24 s vs 50 s for SwiftShader and uses 1 GB less; the recurring "GPU cannot be used" log line is cosmetic. No change. See ../map/objects/host/story-android.
2026-09-25: Android emulators resized for 40 GiB: Play 4096 guest/7168 cap/8000 gate, plain 2560 no-lowram/5120/5900, login 4096/7680/8500, CPU 200%. Play boot 35s -> 24s, guest free 1.1 -> 2.3 GB; both boot+idle pass. story-publisher 152e431. See ../map/objects/host/story-android.
2026-09-25: claude-remote fixed by Angel: unit now starts in trusted ~/claude-remote (memory symlinked to -home-angel/memory, one-line CLAUDE.md), Restart=on-failure/RestartSec=60. Backgrounded session 35d13aa0 at 20:48; no failed user units. Reason: Claude Code 2.1.283 no longer persists home-dir trust and --bg exits without trust.
2026-09-25: Post-RAM-install cleanup. samba-nas: the hard power-offs left 288 NUL bytes in the container's /etc/samba/smb.conf, so Samba 4.12 folded [EmbeddedEra] into [global] (share missing + global valid users broke the anonymous healthcheck). Stripped NULs (backup smb.conf.nul-bak in container), reloaded; all 3 shares list. Scan of text files written since 09-24 found no other NUL damage. codex-remote-control: daemon lost its app-server.pid after the power cut and refused to manage the running server; stopped the orphan, codex remote-control start connected. content-delivery-check: one boot-time "partial" before Postiz was up; manual run complete. claude-remote: Claude Code auto-updated to 2.1.283 at 20:09; --bg now needs workspace trust and home-dir trust no longer persists, so the unit exits. Fix (start it from a trusted subfolder) was blocked by the permission classifier; left for Angel.
2026-09-25: RAM upgrade installed: 40 GiB detected (not the planned 32). Play AVD boot+idle passed in 30 s with 28.5 GiB available. Slots: 8+8+16+8 (DIMM1-4), flex mode, accepted. Open: samba-nas healthcheck returns NT_STATUS_ACCESS_DENIED after boot. See map/objects/host/machine.
2026-09-25: Pre-shutdown check for the RAM install found two orphaned erp-verify-pg-* postgres:17 containers up 5 days since 2026-09-20 11:26. erp-backup names them erp-verify-pg-$$ and cleans up via trap cleanup_pg RETURN, which a SIGKILL skips - the documented background-Bash memory supervisor kill is the likely cause. Removed both. Disk map written to ~/disk-map-20260925.txt; snapraid sync success 03:32, erp-backup 11/11 to R2, rescue archive verified 01:33, all five mounts present.
2026-09-25: RAM upgrade settled. dmidecode -t 17 confirms one 16 GB DDR4 in DIMM3 with DIMM1/2/4 empty, so Angel's 2x8 GB Corsair CMK16GX4M2B3200C16 kit goes in DIMM2+DIMM4 for 32 GB and dual-channel. All modules run 2400 MT/s (Kaby Lake ceiling, no XMP on Dell) - the 3200 rating changes nothing. Trigger was swap at 4.4 MiB free of 4 GiB while memory pressure still read 0.00: no reserve left, next burst hits the OOM killer. Capacity case for a second machine is closed; see map/objects/host/machine and embeddedera-infrastructure-split.
2026-09-23: Android real-note Story previews completed using bounded transient user units; teardown verified and no publishing timer enabled. Runtime details in map/objects/host/story-android, review/approval state in story-publisher-pilot.
2026-09-22: Social complete-day approval and feed-before-Story safeguards deployed (assistant b0653d8); vivid artwork preference saved, WebUSB replacement awaiting review. Existing September 22/23 feed recovery awaits explicit repost-versus-fresh-content decision. Details: content-pipeline.
2026-09-22: Recovered the missed research draft after Angel renewed Claude login. Corrected the generated note's browser-support claims, deployed its review preview, and fixed publish-note to promote the latest saved note rather than its creation commit (assistant main 31c61f5; four isolated Git fixture tests passed). No production article or social post was approved. Details in content-pipeline, SOP-018 v5.8.
2026-09-21: Recovered today's two missing social feeds using the existing approved Postiz IDs after Angel explicitly authorized moving them from September 23. Both published at 18:28 Eastern. Deployed per-day/per-destination coverage warnings and a 15-minute audit with six-hour unresolved reminders; 149 isolated backend checks and phone/desktop browser checks passed. Root cause, remaining content gaps and deployment evidence live in content-pipeline; SOP-018 v5.7 owns the procedure.
2026-09-21: Added ERP location Finished goods / Paul Shop and moved 94 eligible finished assemblies there: Cow Finger 41, Cow Remote 32, Robot Cow 21. Three unfinished Robot Cow outputs remain in In test pending their automated final completion.
2026-09-21: Grafana Stock now groups assembly inventory by location and part number, summing serialized and bulk records into manager-readable quantities. Paul Shop shows Robot Cow 21, Cow Remote 32 and Cow Finger 41.
2026-09-21: Deployed Metabase OSS v0.57.4 at the former management-reporting hostname behind Cloudflare Access for simple management reporting. Four idempotently generated dashboards reuse the canonical read-only ERP SQL; new active assemblies with BOMs appear automatically. Grafana remains available for technical/detail reports.
2026-09-21: Removed the redundant Metabase sign-in from the management path: loopback-only Caddy now redirects the Cloudflare-authenticated root to the protected Build capability public dashboard, while the Metabase application container has no published host port.
2026-09-21: Home's Reports tile now opens the former management-reporting hostname and describes build capability, shortages, inventory locations and attention items. Promoted from assistant sandbox after 542 tests; production commit 62c33e5.
2026-09-21: Replaced Metabase's initial raw technical tables with the requested management layout: four headline totals, one row per assembly, and a focused purchase-shortage table. Live queries returned 94 finished assemblies, four dynamically discovered assembly products and 22 blocker lines; verified at 1200px and 390px without page overflow. Source commit 6d9e24e.
2026-09-21: Added workflow-location inventory to the Metabase management landing page and inventory dashboard. Live readback: Finished goods 0, Finished goods/Paul Shop 94 across three assembly PNs, In test 3 Robot Cow, Quarantine 0. Queries include future Finished goods sublocations and Tested locations automatically; verified at 390/1200px. Source commit a86b3b7.
2026-09-21: Made Build capability interactive: clicking an assembly cross-filters the purchase table in place. Browser-verified Robot Cow changed the URL parameter and reduced 22 blocker rows to its single EE-K-0001 shortage with no failed requests. Source commit 8825029.
2026-09-20: Deployed the Robot-board quarantine and rework server workflow. First-test failures can create unfinished BO-0001 outputs, consume exact pretest material, move to Quarantine, retain every attempt, link a later-readable chip, and release automatically to In test or Finished goods; confirmed Scrap closes the build output and marks it Destroyed. Sandbox release and scrap rehearsals passed. Tester API main d2eb23f; Cow Tester 2.7.0 main 212cb20 awaits the physical Mac pull and bench print/fixture check.
2026-09-20: Staged failed Robot serials 09260042/09260059 for PRECHECK in the live ERP as unfinished BO-0001 outputs 87/114, preserving and attaching all prior failures. No Quarantine cases were pre-decided; the next physical run owns that operator decision.
2026-09-20: Closed the social handoff loop. Publishing a website note now triggers branded feed/Story preparation and a clickable approval notice; a persistent 18:30 timer catches misses and reminds once daily. The single Home confirmation attests the exact image hash and schedules only empty destinations. First production sweep was quiet with zero pending work.
2026-09-20: Reconciled all eight published EmbeddedEra notes with the social ledger and Postiz. Scheduled three missing note packages for September 23–25 at Facebook 09:00 / Instagram 18:00, with four destinations each and no duplicate occupancy. Added an hourly read-only content-schedule-audit.timer that alerts on missing coverage or duplicate risk; first live run passed with zero issues.
2026-09-20: Corrected Grafana System home's stale “Buildable from recorded stock” query. It had not received the all-assembly capacity fix and rendered Robot Cow as a dash because of its consumable coating line. The live panel now uses the canonical query and shows 0 additional from free stock plus 27 ready in the fully allocated active build.
2026-09-20: Grafana capacity now includes “Total buildable now” for every assembly, adding fully allocated active builds to additional unreserved-stock capacity. Live shop-product totals are Robot Cow 27, Cow Remote 0 and Cow Finger 0 from recorded ERP inventory.
2026-09-20: Added a Grafana table listing every inventory line blocking the next additional assembly, including part number and needed/available/missing quantities. Robot Cow's next board is currently blocked by one unreserved antistatic bag; Remote has 18 recorded blockers and Finger has two.
2026-09-20: Clarified EE-P-0002-A in ERP without changing quantity: 270 individual boards = 30 panels × 9, split 243 unallocated (27 panels) and 27 allocated to BO-0001 (3 panels). Part unit is pcs; part description and both stock packaging fields now state nine boards per panel.
2026-09-20: Added live production-test reporting at Grafana /d/ee-test-results. A one-minute user timer mirrors the tester API's SQLite event/serial index read-only into the reports database; Grafana's reports_ro role has SELECT only. Verified the provisioned dashboard and query through Grafana's API. Recent Cow Remote final passes are retained but ERP-pending because their legacy serials have no server-recorded pre-coating PASS/unit.
2026-09-20: Repaired overnight ntfy findings. ERP R2 restore drill now waits through PostgreSQL initialization and accepts growing table counts; restored 10 categories, 22 locations, 278 Kanboard tasks, 142 assistant rows and 14 vault ciphers. Surface probe uses a free port instead of colliding with tester API 8210. Assistant oversized-input canary no longer becomes a promptfoo provider error; targeted canary passed. Store order sync treats durable review states as review work instead of a crashed worker (45 tests and live run passed). Shared user-unit alert names the actual MONITOR_UNIT; 05:41 was KiCad validation, not task digest. Precious logrotate now rotates as root:syslog and recreates angel:angel. Removed parity long SMART test after its hd-idle deferral overlapped Sunday scrub and reached 57 °C; weekly scrub remains surface coverage and drive cooled to 33 °C.
2026-09-18: Closed task #59 after authenticated read-only qBittorrent runtime verification using existing Radarr integration credentials (never displayed). All global limit flags false, limit values -1; all 192 live torrents inherit (-2/-2/-2). No settings changed; temporary session logged out. Evidence: audits/2026-09-18-delegated-task-approvals.md.
2026-09-18: Installed reviewed list-sync from assistant/scripts/list-sync, main 66ceca5. Bounded GET retries, overlap lock, verified import/deletion and retained uncertain records; cron unchanged. Backup: ~/.local/state/list-sync-backups/list-sync-before-2026-09-18. Seventeen isolated fixture tests passed; SOP-033 documents operation. No live mutation tests.
2026-09-18: Enabled user content-delivery-check.timer (15 min, GET-only/no AI) after sandbox and 40 tests; main 00c773d. Private report shown in Content approvals separately from original ledger. First run complete: 2 published / 15 scheduled. Nine queued Stories received reviewed artwork through same-ID updates; no published posts changed. Website-care pilot tooling ready, customer URL pending. Map schedule card and SOP-018/SOP-032 updated.
2026-09-18: Deployed first EmbeddedEra content workflow to assistant main 6c04fc5 after sandbox verification: Story visual-review gate for new approvals, separate draft renderer and read-only publication audit. Existing approved social jobs untouched. Tests:31 passed; 375/1440 browser checks and service health passed. See embeddedera-workflows and content-pipeline; SOP-018 v4.0.
2026-09-18: Home portal tiles now have locally vendored Lucide 1.47.0 SVG icons, recoloured through the EmbeddedEra copper/muted theme tokens and hidden from accessibility APIs so visible labels remain authoritative. Sandbox tests, strict SVG route/traversal checks and a 375 px browser render passed before production deployment (assistant d89c627; sandbox e30b5af). SOP-031 owns future internal-interface/icon changes.
2026-09-18: Built the real SOP-020 Vaultwarden audit path. Installed the official Bitwarden CLI 2026.9.0 user-space and added vault-audit: after one dedicated read-only account bootstrap it logs in by API key, unlocks only in a private temporary runtime directory, matches actual local secret values against decrypted vault contents in memory (item names are fallback), outputs no values, locks and removes the session. sop-audit now exposes SOP-020.key-in-vault; it correctly skips until vault-audit setup is completed rather than claiming a false pass. Offline tests pass.
2026-09-18: Completed hardening item 18/task #59 measurement. qBittorrent has 192 resume records (~2.74 TiB), 154 private torrents, and 153 ratios below 1.0; all inherit disabled global ratio/time limits. Decision: keep share limits disabled—especially all time limits. Found the documented Arr interlock had drifted: Radarr/Sonarr/Sonarr-Anime all had both removal flags true. Corrected each through its local API and verified database values 0|0; no torrent limits were changed.
2026-09-18: Reconciled stale hardening item 16/task #56 from the authoritative 2026-09-11 report: maint-dedupe completed at 08:32, byte-matched and hardlinked 20/20 pairs, freed 140 GiB, reported zero anomalies/failures, and left a clean SnapRAID diff. Updated the hardening map card to done.
2026-09-18: Removed every literal credential from ~/torrentvpn/docker-compose.yml (11 variables, including VPN/Arr/WireGuard/DuckDNS/Vaultwarden/Listmonk/Postiz/Pi-hole), preserving values in mode-0600 .env. docker compose config --quiet passes without recreating containers. Created and pushed private EmbeddedEra/torrentvpn with a deny-all whitelist tracking only Compose and Grafana configuration (commit 26bfd42); staged-file secret scan passed. Moved 17 obsolete compose backups to recoverable desktop trash.
2026-09-18: Made onboard kanboard-sync bidirectional and least-privilege for non-admin users, with the Administration owner bundle retaining all governed boards. Unsupported RPC use from the first worker pass was caught in review and corrected to getProjectUsers. Applied safely: Yara lost Engineering, Finance, and Production access; her Administration membership was deliberately retained because seven tasks there are assigned to her. Post-apply dry-run reports only that blocker and never silently unassigns work.
2026-09-18: Fixed erp-sandbox token: one minted sandbox token is now atomically written to all five ERP_TOKEN_* roles in ~/.config/assistant/env.sandbox, preserving unrelated lines and file mode, then printed to stdout. Verified with a temporary env harness (missing keys, punctuation, mode preservation, no token in logs); no live token was minted during testing.
2026-09-18: Unified Claude and Codex memory. Canonical shared Markdown memory remains ~/.claude/projects/-home-angel/memory/; ~/.codex/memory links to it, and Codex's global ~/.codex/AGENTS.md now requires reading its index and writing durable user/feedback facts there. Codex's application-owned memories_1.sqlite had zero memory rows and was left in place rather than moving a live database. The nightly private-vault mirror and web-publishing exclusion are unchanged.
2026-09-18: Memory/Obsidian/ICM audit passed. notes-sync was clean and current; engineering-notes is private; notes-nightly mirrors Claude memory at 23:55; claude-memory/ is excluded from the web notebook; notebook publishing is healthy on its ten-minute timer. Obsidian Desktop 1.13.7 and icm-architect upstream commit e16cafe6 both match current upstream. Corrected the stale tooling decision that still said ICM had been skipped. The prior session's durable results were already recorded in project/SOP/map notes; raw chat was not copied into the notebook.
2026-09-17: Company social approval hub deployed under Home /home/content; assistant main 2f19279, SOP-018 v2.1. Five packages await approval. Scoped public approved-artwork path supports Meta media fetches; all ordinary app gates remain intact. See content-pipeline for state and scripts.
2026-09-17: Postiz back on at Angel's request — the six containers were already running again, so only the switches were off: removed ~/.config/postiz/DISABLED (social-push was no-opping) and put the social tile back in departments.yml for Administration and Sales. It stays behind the compose social profile, so a plain docker compose up -d still leaves it down; start it with docker compose --profile social up -d. Watch memory: those six hold about 3 GB.
2026-09-17: Git audit and push. ~/assistant (113 commits, no remote) and ~/erp-config (the real home of the print templates; ~/torrentvpn/inventree/templates-src is a symlink to it) both had no remote — now private repos EmbeddedEra/assistant and EmbeddedEra/erp-config, pushed. Also pushed the KiCad library's Nexar partlookup change. ~/torrentvpn is still not under git and should not be until the secrets come out of docker-compose.yml: it carries Vaultwarden's ADMIN_TOKEN, Postgres and Listmonk passwords and a Jellystat token inline, so the file cannot go to GitHub even privately. Everything there is covered by the nightly backup, not by history. FileBrowser already exposes all of it: /srv/home is ~, /srv/torrentvpn is the stack.
2026-09-11: Listmonk added to the torrentvpn stack for the EmbeddedEra newsletter (listmonk + listmonk-db, postgres:16-alpine) — LAN-only by design, never exposed to the internet: 127.0.0.1:9000 loopback only, list.lan via NPM (host id 25) for admin access. Creds in ~/.config/credentials/listmonk.txt (admin login, db password, and an API-user token for list-sync scoped to just the "Engineering notes" list, id 3). Backup: docker-compose.yml.bak.20260911-listmonk. Site side (embeddedera-site-apple: functions/api/subscribe.ts, SubscribeBlock.astro, i18n) writes signups to R2 subscribers/; ~/.local/bin/list-sync (cron candidate, not yet installed — line printed for review, not added) pulls them in every 15 min and deletes the R2 object on success. Blocked on Angel: a Resend API key (or Workspace SMTP) before Listmonk can actually send the double opt-in confirmation — see Notes/projects/growth-plan.md §1.2.
2026-09-10: Server secrets imported into Vaultwarden (Bitwarden JSON import, 12 items, folder "Plex server"); import file shredded. Only copies of ~/.config/{tesla,restic,credentials} are now NVMe + the vault (vault data in docker-backup nightly).
2026-09-10: Vaultwarden LIVE at https://vault.embeddedera.com — Cloudflare A record (DNS-only, 192.168.5.15), Let's Encrypt via DNS-01 (Cloudflare token stored in NPM), NPM proxy host → vaultwarden:80 (websockets, force SSL). First account angelgdo7@gmail.com; SIGNUPS_ALLOWED=false. NPM admin password was lost → new ~/.local/bin/npmpass helper (needs sudo, data dir is root-owned); creds in ~/.config/credentials/npm.txt.
2026-09-10: Added Vaultwarden (self-hosted Bitwarden) to the torrentvpn stack — vault.embeddedera.com, NPM-only, no host ports, data in ~/torrentvpn/vaultwarden/data (nightly docker-backup). Chosen over Bitwarden cloud/1Password (free rule) and KeePassXC (no sync). Map card map/objects/containers/vaultwarden.md. Angel still has to do DNS + cert + first account.
2026-09-10: build-index.sh no longer hard-codes /home/angel (the Mac's silent-data-loss finding) — $HOME, no-op elsewhere; bin/notes-sync is now the single implementation (dry-run + auto message merged in), ~/.local/bin/notes-sync symlinks to it; weekly prompt adopts delete-not-tick; item 17 removed from Open per that rule (closed clean 09-09).
2026-09-09 22:10: bin/notes-sync.cmd added for Windows. Chose PATH-plus-setx over symlinks so Developer Mode is never required; restore.sh stays Mac/Linux-only. .gitattributes added to keep *.cmd CRLF and the extensionless bin/ scripts LF. Not yet run on DESKTOP-1LK2GJK.
2026-09-09 22:00: new-machine setup fixed. notes-sync and part-lookup existed only on the Mac in ~/.local/bin and were never tracked, so restore.sh handed every new machine rules it could not follow. Both are now vendored to bin/ and symlinked in by restore.sh (symlink, not copy, so a vault pull updates them); added a PATH check and split the mangled ponytail/i-have-adhd echo. Verified by running restore.sh on the Mac. Windows symlink support still unresolved.
2026-09-09 16:30: notes-weekly installed (Sun 19:30, Opus, branch + PR gate, usage-limit retry until 22:30). Deadline for the race-week override removal: after 09-14.
2026-09-09 16:15: routes repo created (github.com/angelEmbeddedEra/routes, private) with both Sandling bike courses; both loaded on the Garmin (courses 512961452/512962555) and the Magene C606 V2. Sunday multisport RACE PRACTICE workout 1693043006 with per-section speed/cadence. ~/maintenance/output -> map/reports/ symlink; notes-nightly cron 23:55. Tonight 23:00 badblocks; Fri 08:00 dedupe.
2026-09-09 11:00: parity drill PASSED. Found and fixed the precheck waking the parity disk via SMART (sdb sat at 55 °C for 2.5 h doing nothing); hd-idle cannot see SMART spin-ups.
2026-09-09: history repo is in parity (03:30 sync clean). False "SnapRAID FAILED" push at 23:00 was maint-badblocks's guard matching its own subshell — fixed, alert renamed, installer re-dates timers.
2026-09-08: restic 0.19.1 installed user-space; history repo initialised on the pool; nightly 02:50 + Sunday 05:30 jobs in angel's crontab (duplicate 02:20 precious-sync line removed). Initial backup done 17:03: 27 min, 90 G / 5,399 packs, restore drill PASS. Parity covers it from the 09-09 03:30 sync.
2026-09-08: built ~/Notes/map (15 object cards, 8 process cards); confirmed the WD idle3 fix latched; scheduled hardening items 15/16B/17; installed Obsidian and the icm-architect skill; linked these global rules.
2026-09-14 Grafana landing pages: ee-start (links only) is the org default, ee-home (System folder, headline stat from every report) for Administration, per-team homes via reports-sync HOME_BY_TEAM. Verified through the auth proxy as adiaz (→ ee-home) and yrosado (→ ee-sales, 403 on System/Quality). See grafana.
2026-09-14 Close-out loop live: Review column + close-on-Done in all projects (onboard kanboard-sync), SOP-017 v1.1 published, task-verify every 5 min (first real pass: task 177), sop-tasks and intake write Verify: blocks, sop-audit check renamed closed-unverified, Waiting on you shows human checks. See kanboard.
2026-09-14 SOP-017 v1.2: every task needs assignee, due date, concept, Deliverables + Verify. 86 unowned tasks assigned to department defaults (Sales/Purchasing → Yara, rest → Angel); tasks-deliverables (Haiku, ~$0.004/task) backfills the blocks; audit checks open-unassigned / open-no-due / open-no-deliverables now read Kanboard over the API.
2026-09-14 Intake gained New task (AI scrub to SOP-017 form) and Idea (Ideas project, no due/deliverables). Deliverables backfilled on 121 tasks; 30-day default due on 122. Open: tasks 72 and 128 still lack deliverables (Haiku could not parse them, write by hand).
2026-09-14 Resend quota blown (200 emails: Kanboard per-event mail during the bulk assign). Kanboard email OFF for all; _mailbudget.py (60/day) now fronts tasks-digest, tasks-notify and the assistant mailer; tasks-notify hourly batched per person. SOP-017 v1.4. Books queue no longer shows intake tasks/ideas (intake_kind filter).
2026-09-14 "Waiting on you" moved from home.embeddedera.com to the notebook front page (notebook-build waiting_section: proposals, drafted site notes, pending human checks; notebook-publish also watches those two sources). Home page API kept; approve/reject buttons retired — decide by editing the Status line or publish-note.
2026-09-14 Opening stock count: intake "Stock count" (barcode → OCR → Haiku) + /stock queue confirming into the ERP; SOP-023 published, SOP-005 v1.6 cross-refs it. Sandbox token note: erp-sandbox token only prints; env.sandbox ERP_TOKEN_* must be pasted by hand (to-do below).
2026-09-18: Completed task #176 preventive action from the 2026-09-14 cold-read finding. SOP-015 v1.2 now requires a new-session, non-builder audit every quarter with live Docker network isolation proof, runtime-vs-Compose comparison, filesystem/backup/error-handling checks, independent reproduction, and corrective tasks. The already-scheduled quarterly-pack cron (07:00 on Jan/Apr/Jul/Oct 1) now inserts the checklist and expected Notes/audits/YYYY-QN-cold-read.md path into every pack; dry-run verified the generated section. Updated the server schedule map.
2026-09-14 Stock queue complete: create-part-and-add-stock, New bin, direct Zebra printing (192.168.5.16:9100), label QR = ERP page URL, INVENTREE_BASE_URL fixed to the public host. Resistor row #184 ready for one press.
2026-09-14 First real stock count (22 parts, 19 stock items). Faults fixed: task-verify erp: check used the public host (Access login → cannot tell) → local URL; create-part left orphans when the manufacturer step 403d → reuse same-name part, manufacturer/supplier best-effort, company POST falls back to the owner token; Verify line now follows the chosen part. Deleted duplicate parts EE-C-0156/0161/0168 (no stock). Labels tool live; camera scan in progress.
2026-09-15 Part naming reviewed after the first count: DigiKey-known parts perfect, LCSC-only parts inconsistent → partnorm (MPN decoders + Haiku house-style rewrite + validator) in the stock queue and part-normalise for the backlog. Facilities department + Assign-to picker on New task; Labels tool with scan; item label v2.
2026-09-15 KiCad CI pushed (EmbeddedEra/KiCad cc7d8af). Opening-stock cost: 7 items priced from DigiKey list (estimates, noted), AM90 wrong match cleared, task 210 for Purchasing to price the other 17 from LCSC order history.
2026-09-15 Site-note approval is one press again: notebook Waiting on you links to https://home.embeddedera.com/home/note/ (approve / save edits / reject, behind the gate). Tasks 211 (Nexar key) and 212 (app password on adiaz@, Intake label) raised for supplier data.
2026-09-15 Nexar added as the third part-lookup source (LCSC bridge); vector logo approved and live on the site; decisions recorded: customer data, retention, bank import, one Kanboard, Yara as emergency person; resale certificate facts found in mail (E-595E exists, DigiKey exempt since 2025-05).
2026-09-15 Labels page became Production tools (/tools): print, Move… (Neoden YY1 first), Count. Privacy customer-data live; close-out reporting on dashboard + digest; Nexar lifetime cap 90.
2026-09-15 Project_Template repo created (private template): manifest-check + kicad-checks + fab-outputs; SOP-021/022 drafted; template .kicad_dru files pushed. Follow-up: KICAD_LIB_TOKEN + delete smoke repo (task for Angel).
2026-09-15 KICAD_LIB_TOKEN (kicad-lib-read) set; Project_Template kicad-checks green (starter board checked as a fragment until renamed). Smoke repo deleted by Angel.
2026-09-15 Postiz (social scheduler) deployed behind Access at social.embeddedera.com; cf-host tool created for tunnel ingress + CNAME (SOP-019 v1.1). Provider apps are Angel's to create (task 107).
2026-09-15 Postiz: Google login working (built-in provider = YouTube client; Access-OIDC route abandoned, Postiz sends no state), registration off, public API key stored, social-push built and chained into publish-note. Platform apps (task 107) are Angel's next PC job.
2026-09-16: Postiz disabled at Angel's request (memory: 2.2 GB for the app plus ~1 GB for Temporal/ES idle). Services moved behind compose profile social, containers removed, volumes kept; social-push no-ops while ~/.config/postiz/DISABLED exists; Social tile removed from departments.yml. Available memory went from 5.3 GB to 8.2 GB. Card: map/objects/containers/postiz.md.
2026-09-19: Continued the isolated storefront implementation with Claude workers; new runtime and verification facts recorded in map/objects/containers/embeddedera-store-sandbox and embeddedera-store. Existing production compose/ERP configuration left unchanged.
2026-09-19: Continued the authorized store preview remediation: native B2B onboarding/release browser flow and pending battery listing guard deployed in the dedicated isolated store project. Provider side effects remain disabled. Runtime details live in the embeddedera-store-sandbox map card; project acceptance lives in embeddedera-store.
2026-09-19: Pi-hole was blocking email links (Hilton links.h6.hilton.com via BLP tracking.txt; Chase/JPM/BofA hosts in Aug). Disabled BLP ads/tracking/malware/abuse/redirect + dead dbl.oisd.nl (ids 2,16,24,25,26,33), added HaGeZi PRO + TIF (ids 36,37), allowlisted the Hilton domain. Rule in map/objects/containers/torrentvpn-stack.
2026-09-19: Fixed missing link in the 09:00 Facebook post and tomorrow’s queued feed post; deployed mandatory Facebook website CTA guard (assistant main 8c281ac), preserving Instagram captions. 31 focused checks passed; details and evidence in content-pipeline, procedure SOP-018 v5.3.
2026-09-19: Recovered the missed 18:00 Instagram feed and Story from their original Postiz jobs after a Docker-restart scheduler stall; both published without duplicates. Added five-minute scheduler self-healing/alerts and dual web+worker container health (assistant main 41fc058, torrentvpn 9ecc505); details in content-pipeline, SOP-018 v5.4.
2026-09-20: Diagnosed a Zebra GX430t network-service hang after a six-copy Tools request: TCP connections opened but label payloads stayed unacknowledged and ~HS returned no status. Hardened assistant/zebra.py to require a real status reply, bound the send buffer/timeout, and verify full TCP drain before reporting success; the UI now directs a power cycle and preserves the PDF fallback rather than showing a false print success.
2026-09-21: Deployed the social illustration description/regenerate control after isolated backend/browser/rendering checks. Feature and verification details live in content-pipeline; runtime guard in map/objects/host/claude-tooling.
2026-09-21: Optimized the live Metabase Build capability management report: clearer KPI and quantity names, compact phone cards, color-plus-text status, selected-assembly shortages immediately below capability, purchasing-oriented columns sorted by largest shortage, one workflow-location detail table, card definitions, and a visible ERP timestamp. Robot Cow cross-filter and clear pill verified; 390 px and 1200 px renders had no overflow or failed requests. Source in erp-config/reporting/metabase/.
2026-09-21: Added Finished on hand by assembly to the live Metabase capability table. Current rows reconcile to the 94-unit headline: Cow Finger 41, Cow Remote 32, Dashcam Drive 0, Robot Cow 21. Desktop/390 px renders have no horizontal page overflow; Robot Cow still cross-filters to EE-K-0001.
2026-09-21: Extended the Metabase Build capability Assembly cross-filter to Stock by workflow location. Selecting Robot Cow now shows only EE-A-0002-A: 21 in Finished goods/Paul Shop and 3 In test; the same filter continues to constrain the shortage card, and clearing it restores all products.
2026-09-21: Replaced the Build capability free-text assembly filter with a native searchable Select assembly dropdown populated from active ERP assemblies with BOMs. Options include name + IPN and work by either search term on PC and phone; selection updates the URL, shortage card, and workflow stock card. Existing row-click selection remains. Verified Robot Cow through both paths with no page overflow at 390 px or 1200 px.
2026-09-21: Organized the live Metabase Build capability dashboard into Overview, Build, Locations and Purchasing tabs while preserving its public link. The assembly selection persists across tabs; Overview KPIs, build capacity, shortages and all location cards follow the selection. Robot Cow showed 21 finished and only its own build/location/purchasing rows. Desktop and 390 px checks passed with no failed requests or page overflow; repeat bootstrap preserved tabs. SOP-028 v1.3 published.
2026-09-21: Added one company-wide Current inventory value card to Metabase Overview. Live estimate: $8,356.02, using stock-lot USD cost, positive part base cost, weighted reviewed LCSC history, active USD supplier pricing, then zero when no source exists. It remains global under assembly filtering. Desktop/390 px checks passed with no overflow or failed requests. SOP-027 v1.5 and SOP-028 v1.4 published.
2026-09-21: Audited helping-hands/delegation rules and the Codex worker launcher. Findings and proposed simpler tool-first policy in delegation-rules-audit; no rules or launchers changed.
2026-09-21: Adopted shared proactive parallel AI delegation rules at Angel’s request. Global Claude/Codex instructions, home routers, agent profiles and memory now reference SOP-025 v3.0. No worker launcher or background schedule changed. See ai-capacity.
2026-09-21: Published the scalable Metabase Quality Reporting dashboard with Overview, ERP Data Quality, and Inspection & Testing tabs. Twenty-one read-only checks cover duplicates, missing part data, BOM readiness, workflow queues, serials, and test setup; Product and Severity filters and direct ERP links are included. All 13 cards executed successfully through the public Metabase API, repeat bootstrap preserved dashboard ID/link, and SOP-016 v1.3 documents use and the current no-yield limit.
2026-09-21: Published the former Home reporting hub in the actual Assistant-backed Home service after finding the earlier Pages deployment was not the live origin. The Home Reports tile now opens phone-first cards for Management Reports, Quality Reporting and Technical Reports; cards come from one JS list for future expansion, preserve ee-theme, and mark the last-opened report locally. Assistant sandbox route and full suite passed (577 tests, 31 subtests); production commit 4c76f3c, origin route and no-store header verified.
2026-09-21: Installed user ai-work-runner timer for authorized background draft/review jobs, with capacity/scope checks, Landlock isolation and notebook results. Source ~/ai-work-runner; runtime documented in the Claude tooling and schedules cards.
2026-09-21: Started isolated linked-Story publisher prototype; see story-publisher. Read-only KVM/memory inspection only; no host services or schedules changed.
2026-09-21: Consolidated published reporting at reports.embeddedera.com: a phone-first Metabase overview now links stable Management and Quality routes; Home and old management URLs redirect compatibly. The tunnel moved from Grafana port 3000 to Metabase proxy port 3002. Grafana remains healthy and preserved on loopback only. Legacy Grafana bookmarks now redirect to the closest active area instead of failing. Source commits erp-config 6854884/70cc3c2 and assistant 10f474c.
2026-09-22: Installed isolated, bounded Android emulator/Appium rehearsal on this PC at user request; native sample-app control verified. See story-publisher and ../map/objects/host/story-android. Existing services and schedules unchanged.
2026-09-22: Completed the clean reporting cutover. Removed the retired management hostname from Cloudflare Access, DNS and tunnel ingress; removed the old Home /reporting/ route and Grafana bookmark redirects; updated current SOP links. reports.embeddedera.com is now the sole published report address. Grafana remains local-only. Source commits erp-config 1966bd0 and assistant c0c4ba6.
2026-09-22: Verified 16 GiB RAM inventory and documented Dell upgrade limits; slot occupancy requires sudo read. Offline media/link fixture passed; official Play Store reached sign-in but low-memory runtime is not production-ready. See story-publisher and machine card. No accounts, publications or existing services changed.
2026-09-22: Read-only memory audit found 70 running containers (~10.15 GiB grouped Docker working-set readings), led by sandbox ERP 2.36 GiB, Postiz 1.99 GiB and production ERP 1.67 GiB. Saved plex-memory-plan with sandbox-first optimization, workload/ownership gates and RAM purchase reassessment. Planning only; no service/configuration changes. User terminal DIMM output not yet visible.
2026-09-22: User supplied DIMM inventory: one 16 GB DDR4 UDIMM in DIMM3, three empty slots, running at 2400 MT/s/1.2 V. Updated machine card and plex-memory-plan; adding a compatible 16 GB module is the proposed 32 GB path. No hardware or service changes.
2026-09-22: Owner requested ERP sandbox off except during testing. Stopped five sandbox containers without deleting volumes; disabled/stopped development-store ERP sync and observation timers while idle. Shared memory and sandbox-workflow now require startup for tests and cleanup afterward. Production ERP unchanged.
2026-10-11: codex-remote-control failing every minute since 16:42 ("app server is running but is not managed by codex app-server daemon"). Root cause: the unit is Type=oneshot with the default KillMode=control-group, so systemd killed the daemon-spawned app server as soon as codex remote-control start exited; the Codex desktop SSH connection then booted its own unmanaged app-server --listen unix://, which blocks the daemon. Fix: added KillMode=process to ~/.config/systemd/user/codex-remote-control.service, killed the idle unmanaged server (pid 1348529), daemon now running, timer green.
2026-10-13: Installed figcraft 0.5.1 (CLI + user-scope MCP, no folder limit) on the Mac with a figcraft-only run of ~/Notes/bin/craft-setup. Fixed craft-setup for macOS: BSD head rejects head -n -2, which aborted the old-version cleanup under set -e; replaced with a portable awk.
2026-10-13: Installed FigCraft.app 0.5.1 on the Mac from the release .dmg (SHA256 verified, notarized Developer ID, Learning Machines LLC). craft-setup does not install the Mac GUI apps; it only covers the CLI and MCP server, so the .app does not auto-update.
2026-10-14 episodic-memory installed (obra/episodic-memory 1.6.0, MIT, plugin from superpowers-marketplace): local semantic search over Claude + Codex transcripts. Settings in ~/.claude/settings.json env: EPISODIC_MEMORY_SKIP_SUMMARIES=1 (no Claude/Codex calls; summaries are display-only) and EPISODIC_MEMORY_CONFIG_DIR=/mnt/storage/claude/episodic-memory (pool, not root; SnapRAID already excludes *.sqlite*). First sync 10:05-11:07, 4,161 sessions, 3.2 GB. Syncs in the background at every session start (SessionStart hook). Test: "why did the story not publish after tapping share" found the 09-23 swallowed-tap passage that the desktop app's transcript search (desktop sessions, substring only) missed. Gotchas: the plugin cache is managed, so never npm install in it by hand; its MCP wrapper installs deps on first start. Per-machine only (no sync); the vault stays the cross-machine memory. The archive is another plaintext copy of every transcript.
2026-10-14 Pi-hole: exact allow clarity.microsoft.com (Clarity dashboard; hagezi pro list blocks it). www.clarity.ms was also allowed: the dashboard calls it, and with it blocked the "New project" button never rendered (phone 192.168.4.109 hit it 22 times). Consequence: our own visits from home are recorded too; exclude them in Clarity by IP filter if they pollute recordings. Pi-hole v6 rejects some characters in --comment (an en dash failed); use plain ASCII.
2026-10-21 CLOUDFLARE AUDIT + HANDOFF (nothing changed yet; read-only API calls only). Owner said "do all recommended"; chat hit the 120k budget so work starts in a fresh chat. Setup read: zones embeddedera.com + oddlyfittinggoods.com (Free), Access 31 apps, tunnel plex, Pages x10 (no Web Analytics on any), Workers ee-share/ee-watchdog, R2 x2 (90d lifecycle), D1 x3, KV ee-watchdog, cache rules x3 for the store, tiered cache on, ssl=full, HSTS off, DNSSEC disabled, no custom WAF/rate-limit rules, Bot Fight Mode off, DMARC p=none on both domains (oddly has no rua). Creds: ~/.config/cloudflare/env (token lacks Gateway, RUM, Registrar, firewall/rate-limit read; WAF write untested). NEXT, in order: (1) store 504s: GraphQL (httpRequestsAdaptiveGroups, host store.embeddedera.com) shows ~36 504s/hour all day 2026-10-20, 637 on / and 332 on /wp-login.php, cacheStatus miss, plus 330 400s on / from curl/8.5.0; store is 200/302 when probed by hand and all containers healthy, so find what prober hits / and /wp-login.php about every 100 s (health timers, ee-watchdog worker, uptime check) and why the tunnel times out; (2) WAF custom rules (block xmlrpc.php, .env, wp-config probes; apex had 1301 404s/day) + 1 free rate-limit rule on store cart/checkout/account; (3) ssl full -> strict (proxied records are tunnel/Pages, vault.embeddedera.com is DNS-only 192.168.5.15 so unaffected), then HSTS without preload; DNSSEC needs registrar DS (registrar unknown); (4) Web Analytics on the Pages projects (needs token RUM scope, ask owner once); (5) Bot Fight Mode on, AI crawler view only (do not block, site wants traffic); (6) DMARC: add rua for oddlyfittinggoods.com, read reports 2 weeks then quarantine; (7) Turnstile on store login/register/contact (confirm token scope; my path 7003 no-route was probably the wrong URL); (8) Notifications webhook to ntfy for tunnel down / Pages deploy failed (could not read existing policies). Skip: Email Routing, Gateway/WARP, Queues/Workflows/Vectorize, paid add-ons. Store changes touch production: follow decisions/sandbox-workflow.md, update map/objects/host/cloudflare-access.md and the store cards, run map/_meta/build-index.sh, then notes-sync.
2026-10-20 CLOUDFLARE AUDIT step 1 DONE (read-only): store 504s are NOT an outage and NOT our prober. GraphQL shows every 504 has userAgent nginx-ssl early hints, originResponseStatus 0 (never reached the origin): Cloudflare's own Early Hints cache fetches (zone setting early_hints = on; CF docs warn of 504s with RequestSource earlyHintsCache). ee-store-prod-health (every 5 min) logs 200 on / all day. Decision: leave Early Hints on, ignore these 504s in future audits (filter that UA). Step 2 BLOCKED: PUT to http_request_firewall_custom entrypoint (zone embeddedera.com; no existing ruleset) was denied by the auto-mode classifier as a shared-resource change; nothing written. Draft rule (block): path eq /xmlrpc.php, contains /.env, /.git/, wp-config, eq /wp-content/debug.log; store code only mentions xmlrpc in production-guards.php (a guard, check it before enabling). Needs owner OK or a Bash allow rule for curl -X PUT to api.cloudflare.com, then steps 2-8 continue as listed above.

Sandbox follow-up¶

Audit installed ~/.local/bin/assistant-test isolation: daily canary uses port 8202 and main assistant environment despite documentation claiming sandbox 8201. Do not run it as a sandbox test without verifying provider/ERP isolation.
2026-09-22: Added user-requested on-demand ~/android-login phone browser setup command. Local transient service and user-space dependencies, no root installs or boot schedule. Administration-gated android.embeddedera.com uses existing tunnel and localhost6089, with origin JWT checks. ERP sandbox remains stopped. See story-publisher, ../map/objects/host/story-android and SOP-041.
2026-09-22: Completed unpublished Instagram/Facebook native Story link composition checks on bounded headless emulator; drafts discarded and runtime stopped. Play headless cap now4096MiB/start4800MiB, browser profile unchanged. See story-publisher and ../map/objects/host/story-android. No production posts or service changes.
2026-09-22 evening: Owner split Story ownership from feed pipeline. Eight queued Stories canceled; WebUSB feeds scheduled September 23. Replaced daily research cron with bounded hourly seven-day content-supply refill; approval gates retained. Assistant a250ec1/44044fa/b59cf33; details and initial refill status in content-pipeline.
2026-09-28: erp-backup verify now sweeps erp-verify-pg-<pid> scratch containers whose PID is dead before starting its own (a SIGKILLed run skips the RETURN trap); tested with a dead-PID and a live-PID container.
2026-10-01: New invoice tool built on the sandbox (assistant-sandbox branch sandbox, erp-config branch sandbox-invoice in worktree ~/erp-config-invoice). Evidence in audits/invoice/. Not promoted; waiting on owner's INVOICE_* bank values.
2026-10-01: invoice review fixes landed on the sandbox branches (assistant f13c43b, erp-config sandbox-invoice e3d5e69). Bank block now lives in an ERP report snippet installed by assistant/scripts/invoice-bank-snippet.py, not in order metadata. erp-sandbox up guard added (see sandbox-workflow).
2026-10-01: All outgoing mail moved from Resend to the Google Workspace SMTP relay (owner-approved): invoices from billing@, tasks digest/notify + Kanboard from tasks@, assistant mail from hello@. App password of adiaz@ in assistant env + kanboard mail.env. Resend kept as a no-duplicate fallback in _mailbudget/mailer until ~2026-10-08, then remove the key, the fallback code, env[RESEND_API_KEY], and close the Resend account.
2026-10-01: ICM consolidation plan drafted (artifact https://claude.ai/artifact/3GxcqFhxN3RX9xEx8Km7dT); inventories saved in audits/icm-inventory-2026-10-01/. Awaiting owner decisions before Phase 0 (path registry, vault-lint warn mode, per-file move map).
2026-10-01: ICM plan v2 (same link): 5 areas Sell/Make/Engineering/Admin/Systems + Oddly separate + Personal; Home cockpit (area pages generated from vault labels, one search bar, forms over engines, settings pages); headless proven engines (no home-made DBs), ERPNext trial decides engine; Kanboard kept during move then retired; statements under Admin→money, importer after CPA hour. Open: CPA hour, start Phase 0 + freeze.
2026-10-01: Operating model confirmed by owner: build to stock (shop has run this way for ~2 years); orders ship from finished stock, stock targets drive builds; build to order only for custom work. Workflow diagram + ICM plan updated.
2026-10-01: Stock locations confirmed: Neoden YY1 / Shelf A + Finished goods / Paul Shop (consignment, invoice consumes from it); restock Paul via the generic Move stock form; proposed un-nesting Paul Shop from Finished goods (sandbox first, not done).
2026-10-01: CPA hour parked by owner (not now). Overnight build started: cockpit on sandbox, jobs page, NOW.md, vault-lint warn, path registry, move map, Sell pilot prep.
2026-10-01 ~22:30: Owner authorized going LIVE overnight ("you can go live… use the sandbox for your testing… keep the inventory, all the part numbers working… make sure we can always go back"). Rollback points: git tags pre-cockpit-20261001 (assistant main 0140145), pre-icm-20261001 (Notes 07f31f6, ~/.local/bin); ERP backup 2026-10-01_1413 + fresh one before any ERP change. Stale sandbox WIP parked on branch wip/daily-story-handoff-20260923.
2026-10-01 23:20: Home cockpit LIVE (assistant main 180cfef; rollback tag pre-cockpit-20261001): areas from _system/areas.yml, search (SOPs, actions, ERP parts/customers/contacts/orders, jobs, projects), Needs-you strip, Jobs view (131 jobs), health, Move stock form (/stockmove). Generators jobs-index/now-build every 15 min (cron-index), vault-lint warn-only in notes-sync. Sandbox ERP proved Paul Shop un-nest + move/undo; production un-nest blocked by safety check, left for owner. Move map (437 rows), Sell pilot, memory split in audits/icm-inventory-2026-10-01/.
2026-10-02: Owner chose Tabler (web app shell) over Flutter, layout 1 'Workshop' (vertical dark sidebar, copper on dark). Building at /app beside old /home; installable (PWA manifest); Capacitor later only if native scanning/printing needed. Keep Cloudflare Access + Google.
2026-10-02 00:45: New app LIVE at home.embeddedera.com/app (assistant 7c55b10): Tabler Workshop shell; KPIs verified live (unpaid $54,750 SO-0001, Paul Shop 94 units, Finished goods 2). Old /home unchanged. Found: admin ERP token gets 403 reading sales-order metadata → invoice list for admin users may fail; KPI uses sales token.
2026-10-02: Owner review of /app: area pages add no value; New invoice + Move stock good; PARK Intake, Stock count, Books, Ask the assistant; hide ERP from users; Production tools needs work. Redesign: menu by objects (Sell: Invoices, Customers; Make: Parts, Assemblies & BOMs, Build orders, Stock, Purchase orders) with list/detail/New. Next forms: New part, New assembly + BOM editor, build start/finish, PO, receive, ship. Spec from SOPs → audits/icm-inventory-2026-10-01/forms-spec.md.
2026-10-02: FIXED live bug: next-IPN ignored revision-suffixed numbers (would have issued EE-A-0002 while EE-A-0002-A exists; ERP allows duplicate IPNs). forms.py (assistant 57e913b), erp-part + kicad-erp-sync (ops 886d9e6). Next: A-0010, C-0215, P-0007, K-0007. Forms spec: audits/icm-inventory-2026-10-01/forms-spec.md (19 conflicts, 12 questions). Finding: nothing consumes Paul Shop stock on invoice today; SO-0001 has no allocation/shipment.
2026-10-02: Owner decisions for forms: IPN suffix revisions (EE-P-0006-B); BOM edits Engineering only; part image warn-not-block; invoice to Paul ships billed units from Paul Shop in the same step.
2026-10-02: /app lists + detail pages LIVE (assistant ea69a7e): Sell (invoices, customers), Make (parts 230, assemblies 8 w/ BOM + can-build, builds, stock, POs); parked items removed. Open: production ERP token gets 403 on purchase orders. Marketing page approved in full (website, search ranks, social, leads, pipeline) — building with daily collector marketing-snapshot.
2026-10-02: New part (/partnew) + Assembly/BOM editor (/bom) LIVE (assistant 7432b61); proven on sandbox ERP. Undo of a new part deletes only if unused (falls back to deactivate). Number of an undone part can be reissued (harmless: never used).
2026-10-02: Next after Marketing: Unit page (#unit/, device history: timeline built→tested→moved→invoiced, test results from tester, location, built-from, repairs) reached by typing a serial in the top search. No camera scanning for now (owner). Then forms: build start/finish, PO, receive, ship/invoice Paul from Paul Shop.
2026-10-02: Marketing page LIVE (sell → Marketing; collector marketing-snapshot 08:05 daily). Not connected: X (paid API tier needed — leave off), Pinterest (needs OAuth token), quote-source attribution (form posts only to Web3Forms). Stock grouped by part+location with serial chips. New part requires Minimum in stock.
2026-10-02: Owner chose ACTUAL cost (weighted avg purchase price of stock on hand; fallback preferred supplier at usual qty). Plan queued: assembly Cost tab (+missing-price flags) → labour/overhead settings → Margins page (Money section, admin only). Needs from owner: shop hourly rate, minutes per Robot Cow. Overview redesign proposed (needs-you merge, 6-number strip, Make/Sell/Systems cards, Set minimums screen).
2026-10-02: Margins = parts cost only; labour assumed free (owner). No labour/overhead settings.
2026-10-02: FIXED live: admin users got InvenTree 403 opening invoices (admin token can't read sales-order metadata) → invoices.require_invoice now always uses the Sales token after the Sales/Admin gate (assistant 01a946f). Mark paid exists on /invoice (recent list → Mark paid → method/date/amount/ref → Record payment); app invoice detail to get the same button with the Overview frontend.
2026-10-02: LIVE (assistant 193a391): stock location tree + long-press bulk move + location create/edit/delete (system locations locked); Unit page + serial search; Overview v2 data; /minimums (Set minimums + Paul Shop targets in Notes/_system/settings/stock-targets.yml, Paul Shop EE-A-0002-A: 20). In progress: Overview v2 UI + invoice Record payment in app; actual-cost + margins backend.
2026-10-02: Overview v2 LIVE (assistant e54246e) + invoice Record payment/PDF/Email in app. Cost/margins API committed (ccb9d63, not yet deployed). Real parts cost: Robot Cow $11.03 (29/32 priced), Cow System $23.91, Roper $23.91, Remote Set $12.88; margins ~98% but overstated — 46 parts unpriced (2 PCBs EE-P-0004-A/EE-P-0006-A, consumables EE-K-0003..0006, 40 components). Stock UI: owner wants chips for all locations, no dropdown.
2026-10-02: Owner decided invoice reminders: schedule 3 days before due, on due date, +7, +14, +30 (then stop email → call task); FULLY AUTOMATIC from day one; LHG/Paul included like everyone. Best practices: check balance right before send, partial → remaining amount, weekday mornings from billing@ cc billing@, PDF attached, one per stage (idempotent), per-customer/per-invoice pause switch, logged on the invoice, no late fees (terms don't state any).
2026-10-02: LIVE (assistant 1ceca5d): stock location chips (no dropdown), assembly Cost tab, Money→Margins (admin), pull-to-refresh in home-screen app. In progress: invoice reminders (dry-run default; templates + schedule editable on /invoice).
2026-10-02: Invoice reminders LIVE in code (assistant 756bc34), mode=dry-run (~/.config/assistant/invoice-reminders.mode). Cron hourly 6-18, sends once/day at configured hour. Dry run: SO-0001 → courtesy Mon 10-12, due 10-15, +7 10-22, +14 10-29, final Mon 11-16, call task 11-17. Switch: echo live > that file. Settings + templates on /invoice.
2026-10-02: App organization rule (owner): menu lists things; actions live on the page of the thing they change. Minimums move to part page (inline) + Parts → Minimums tab + Below-minimum filter; Paul targets → Stock › Paul Shop › Targets; 'Set minimums' leaves the menu.
2026-10-02: LIVE (assistant 0b764ab): minimums on part page + Parts→Minimums tab + Below-minimum filter; Paul targets under Stock›Paul Shop›Targets; 'Set minimums' out of menu; pulltorefreshjs 0.1.22 (MIT) + silent auto-refresh.
2026-10-02: Workflow catalog in Notes/_system/workflows/ (27 workflows by area; orphans maint-badblocks/drill/dedupe → retire, pod-pins → pause). Fixed jobs-index under cron (missing XDG_RUNTIME_DIR dropped 27 user timers; ops 59662d6). Oddities logged in _index.md: precious-verify cron OR bug (runs every Sunday + days 1–7), erp-output-clean every minute vs hourly docstring, 3 jobs without cron-run wrapper, sysstat/anacron duplicated.
2026-10-02: Owner wants delete/edit on records (stray LCSC PO-0002: Pending, 0 lines). Building lifecycle actions (PO delete/cancel/edit, build cancel, part obsolete/delete-if-unused, stock scrap, customer/supplier archive) with preview/confirm/undo; UI = '⋯' menu + long-press/right-click on list rows, after the scroll/sheet fixes ship.
2026-10-02: Owner rule: every click in /app stays inside the app (tasks, projects, SOPs, engineering, images). Asked for part/assembly pictures, in-app projects (notes), in-app tasks (Kanban board).
2026-10-02: App roadmap moved to embeddedera-app (its own project note). Healthchecks.io stays as the single external dead-man alert.
2026-10-06: job failure handling built (SOP-043, ../map/processes/job-failure-handling): cron-run hook + OnFailure + 10-min sweep, unchanged re-run of transient failures (max 2/day, deny-list in assistant/job_failure.py), one alert + one fix task per incident, job-incidents, Monday report. No AI, no code changes. Released v2026.10.32+.
2026-10-08: Alert sweep. (1) cron-watch pod-pin-feeds limit was 14h for a 09:45/18:45 job (15h gap) → 17. (2) story-preflight paused Instagram on one uiautomator "Native hierarchy unavailable" flake → rehearsal now retried once before pausing (story-publisher 9e7c13b). (3) surfaces-check still looked for the old home tiles (tilesOurs) after home became the app → markers id="sidebar"/id="view". (4) content-buffer-fill skipped ~20h as "capacity unmeasurable": the Claude CLI usage snapshot only advances while Claude is in use → accepts a reading up to 6h old when session <50% and weekly <80%, and logs claude-usage failures. Real content stall: the 3 ready notes failed independent review and nothing revises them, while weekly-research counts scheduled notes against the 7-draft cap (deadlock). (5) engineering-deploy has been failing since 2026-10-06: the AFEX history rewrite left the build mirror ~/engineering/repos/Cow_Robot_Board diverged (local-only commits all exist upstream by subject); changing clone_or_pull to fetch+reset was blocked by the classifier, left for the owner.
2026-10-08: Audit pass 2. Emulator transient unit ended "failed" on every normal shutdown (emulator SIGKILL after its 20 s grace) → SuccessExitStatus=SIGKILL (story-publisher). kicad-check: block .kicad_pcb files have no Edge.Cuts, so invalid_outline is now fragment-exempt (110 → 34 errors); remaining are real: 2 LED footprints without 3D models (Kingbright KPA-3010, WS2812B-2020), USB_C block clearance/hole_clearance (both copies), NRF24_smd ERC 2e. job-failure-report seeded (was "never run", weekly Mondays).
2026-10-08: engineering-deploy green again (failing since 2026-10-06). Owner reset the build mirrors in ~/engineering/repos to origin after the AFEX history rewrite: Cow_Robot_Board, Boost_Gauge, PSU, Remote_Universal, Universal_Tester_003, and Cow_Remote (also left checked out on branch firmware-release-manifests). Gotcha: any future force-push/history rewrite needs the matching ~/engineering/repos/ reset in the same change, or engineering-build's pull --ff-only stops every deploy. All job incidents closed.
2026-10-08: Content writer/reviewer/revise moved onto the Max plan's $100/month API credits (app v2026.10.35). ~/.local/bin/api-credits keeps a per-job ledger (~/.local/state/api-credits/ledger.jsonl) and monthly shares in ~/.config/api-credits/budgets.json (cap 90, reset day 8 assumed); one ntfy per job per cycle when a share runs out. The Max-subscription capacity guard no longer gates content. Decision: ~/Notes/decisions/api-credits-max-plan.md.
2026-10-08 night (owner asleep, asked for unattended improvements): saved 4 shared articles + index ~/Notes/reference/README.md (memory points to it); "show the receipts" rule in worker/engineering agents; reading-weekly job (Sun 08:00, cron, map card processes/reading-weekly.md, backfilled 2 articles, $0.41); app v2026.10.36: job alerts carry likely cause/fix, frequent deny-listed jobs defer one transient failure (false 11:02 ai-capacity-refresh page), post-image zero-checks before AI review, api-credits ledger model/task + passed/failed with $/pass; cron-watch limits computed from the crontab (9 hand entries removed); precious-verify crontab line fixed to really run first Sunday only (was every Sunday; backup ~/.local/state/crontab.bak.2026-10-09-precious). Cricket ad v3 on branch ad-v3 (not merged/posted). Off-box watchdog built (~/embeddedera-watchdog) but not deployed: Cloudflare token lacks Workers/KV/DNS edit. Job fixer not built (prior classifier block). Stopped when the classifier began denying routine reads.
2026-10-10: offsite-backup stopped at the R2 9 GB stop line (9.29 GB, all in bucket embeddedera-backups). Cause: erp-backup's r2_prune never deleted anything since it was added (log 'kept 49 sets, removed 0' nightly): the stamps were passed with a here-string AND the script with a heredoc, both on stdin, so the script read an empty list. Fixed (stamps via STAMPS env; backup ~/.local/bin/erp-backup.bak.20261010). Dry run on the local list: 49 sets -> 16 kept, 33 dropped (~5 GB freed). The fix runs with tonight's 03:15 erp-backup; offsite-backup resumes 05:15 on 2026-10-11. Expected R2 after: ~5.5 GB of 10. Checked ~/.local/bin for the same pattern: none.
2026-10-10: job-fixer patched after it missed the R2 incident (it only saw offsite-backup's own log): read-only access to ~/.local/state logs + ~/Notes/map, 'the failing job may be the messenger' brief, per-pass reasoning saved in ~/.local/state/job-fixer/notes/, no-diff = diagnosis ntfy + stop (no 3x spin). Replay on the same incident: maker found erp-backup r2_prune's double-stdin bug from 'kept 49 sets, removed 0' and wrote the same fix; the separate checker REJECTED the PR because nothing tested a change that enables deleting backups (correct caution). Incident marked done; fix had been applied by hand.
2026-10-10: job-fixer now requires a keep/delete test for any destructive fix (maker writes it; checker rejects without it). Added ~/.local/bin/test_erp_backup.py for the r2_prune fix: extracts the real function, stubs R2; passes on the fix, fails on the old bug (0 deleted).
2026-10-10: Script intent audit (54 scheduled ~/.local/bin scripts, 3 auditors + adversarial verifiers): 27 confirmed bugs, 1 rejected (vault-audit skip is deliberate). All fixed, originals *.bak.20261010b. Highlights: offsite-backup over the stop line now still runs forget/prune (could never self-recover); job-fixer test pipe kept exit status (bash -o pipefail) and budget-skips no longer burn tries; erp-backup store-verify assert updated for the public store (would have failed every Sunday); cron-watch lateness now from the last SUCCESS (.watch-last-ok.json; a job failing every run never went LATE) without double-paging failures; quarterly-pack reports the quarter that just ended when run in its first 7 days (Q4 pack on 10-01 was all zeros), SO COMPLETE=30 + shipment_date; surfaces-check now checks