Roles and skills — open gaps, what we reuse, which model

Part of roles-and-skills. The twelve gaps the plan did not yet answer, the reuse/skip list, model choice and cost. Text moved verbatim from roles-and-skills; references such as "above", "below" and "earlier" point into that note or its sibling files.

What this plan does not yet answer

Written down so the gaps are visible rather than discovered halfway through a build.

1. Where the words go, and what must never be in them. The assistant runs on someone else's model, so every message and every record it reads leaves the building. Customer names, part numbers, prices and quantities are commercial data going to a third party. That may be perfectly acceptable — it is the same class of thing already in Gmail and Cloudflare — but it should be a decision, with a short never-paste list beside it: vault contents, bank details, the age key, anything an employee should not have in the first place. Also decide whether a local model on the server is worth it for Finance later, where the data is most sensitive.

2. Cost, and what happens at the cap. No budget anywhere in this plan. Needs a monthly ceiling, a per-department view of who is spending it, and a defined behaviour when the ceiling is hit — stop serving, or warn and continue. Same discipline as the distributor quota, and for the same reason.

3. It fails the moment the house does. Chat, ERP and gate all sit on one machine on one connection. If she works this way daily, an outage stops invoicing, not just browsing. Either accept that and write the fallback into the SOP — raise it in the interface later, or on paper — or revisit the hosting question, which is already open.

4. Wrong but plausible. The dangerous failure is not a refusal, it is a confident invoice for the wrong customer with a similar name, or last month's price. Rules needed: never choose between two near matches, always ask; never invent a price; show the resolved order and the customer's full address on the confirmation; and a stated way to correct a mistake after the fact, including what happens to the consumed order number.

5. Nobody has said what "working" means. Without a measure, this expands on enthusiasm and survives on inertia. Something countable: invoices raised through chat versus the interface, how often a draft is corrected before sending, how long it takes end to end. Reviewed after a month, with killing it a real option.

6. Skills need document control too. They are procedures with an audience of one machine. Same treatment as SOPs: version, status, revision history, an owner, and one hard gate — a skill cannot move from draft to active until its red-team cases pass. Otherwise the safety work rots the first time someone edits a skill in a hurry.

7. There is nowhere safe to test. The red-team suite would run against the live ERP, which is the only instance that exists. Needs either a throwaway instance from the nightly backup, which is already proven to restore, or a strict convention of test customers and parts that the suite creates and removes. The first is better.

8. Language. The website has a Spanish version because customers need it. If anyone here would rather work in Spanish, the assistant should answer in Spanish, and the confirmation screen should too. Cheap to do from the start, awkward to retrofit.

9. Who watches the log. An audit trail nobody reads is decoration. Someone reviews denied attempts and unusual actions weekly, or the daily digest goes to Administration, or both.

10. Handoff. When the assistant cannot do something, she needs one action that sends the whole thread to Administration, rather than a dead end and a text message.

11. The new pieces are not in the backup. departments.yml rides in the vault, but chat history and the audit log are new state that nothing backs up yet. They are also the records that matter if a dispute ever turns on who authorised an invoice.

12. Two people, one order. Concurrency is unlikely at two people and inevitable at five. Decide now whether an order being edited is locked, or last write wins with a visible warning.

What we reuse, and what we deliberately write ourselves

Checked 2026-09-12 against the projects themselves, not blog posts.

Use

Piece	What it is	Why it fits
Claude Agent SDK (claude-agent-sdk)	the agent loop, sessions, tool allow-list, hooks	allowed_tools restricts the agent to a literal list in code, and a pre-tool hook can deny a call outright — that hook is the confirmation gate. The scheduled routines here already run on the same machinery
Open WebUI (BSD-3, active)	self-hosted chat interface	the only candidate with clean trusted-header auth, which is exactly our gate: Access verifies Google, passes the identity, the UI logs the person in. It can also forward that identity into our tools
promptfoo	adversarial test suite as YAML, runs in CI	custom assertions can check both the assistant's answer and the HTTP status our action endpoint returned — the "must fail twice" rule becomes a config file
Langfuse (MIT, self-hostable)	tracing and conversation logging	gives the log table, the per-turn cost, and the "which drafts were corrected" question without building any of it
Our own thin action service	the five allowed actions	this is the part that must be ours; see below

Skip, with reasons

• MCP for this. A read-only inventree-mcp exists in the official InvenTree organisation, but it cannot write, so it cannot raise an invoice, and it has almost no usage yet. For five fixed actions, plain functions behind our own service are simpler to audit. MCP earns its place later, when several data sources or several agents need the same connector.
• Guardrail libraries. Rebuff was archived in 2025 and LLM Guard in 2026. NeMo Guardrails and Guardrails AI are alive but bring a validation language for a problem a fixed JSON schema and server-side checks already solve, more auditably. Honest verdict: none of them earns its place here.
• LangGraph, OpenAI Agents SDK, smolagents. Wrong shape or wrong vendor: a graph state machine for one skill, a tie to another model family, or an agent that writes and runs code when we specifically want structured calls.
• Chainlit. Human-in-the-loop has been requested for years and is still not built in.
• Any WhatsApp bridge built on a logged-in phone session. Against Meta's terms and the number gets banned. If WhatsApp happens, it goes through the official Cloud API.
• Adopting a whole "chat to your ERP" project. None exists for InvenTree that is production-ready. Nothing to adopt.

The part that must be ours

The action service — resolve the customer, resolve the part, price from the price list, validate the quantity, build the request, return a pending action for confirmation. It is a few hundred lines and it is where every guarantee in this plan actually lives. No framework can supply it, because it encodes our rules. Buying a framework to avoid writing it would move the guarantees somewhere we cannot inspect.

Likewise the skill files. A skill per department, versioned, citing its SOP, is the thing only we can write, because it encodes how this shop works.

Which model, and why the cheap one is the right default

The job is not open-ended reasoning. It is turning one sentence into a structured request — customer, part, quantity — against a fixed list of actions, where the server resolves every field and a person approves the result. That is a small-model job.

Default: Haiku. The cheapest of the current family and easily capable of extraction plus tool calls. What makes it safe here is architectural rather than a judgement about the model: it never decides permissions, never sets a price, and never sends anything. A weak answer produces a wrong draft that the confirmation screen catches, or a clarifying question. Cheap failure, not dangerous failure.

Escalate to Sonnet, per skill, on evidence. Move a skill up only when the log says to: the small model asks for clarification too often, its drafts get corrected too often, or a skill genuinely needs several steps of reasoning — reconciling a purchase order against a packing slip, say, rather than raising an invoice. Escalation is a line in the skill file, so it is versioned and reversible.

Opus and above are not for this. They cost several times Sonnet and add nothing to structured extraction. Reserve them for design work, not for the daily chat.

A middle path worth having: retry, then escalate. If the small model's output fails validation twice on the same request, retry once on the larger model before giving up. Most sessions never trigger it, and the ones that do are the ones worth the money.

Keeping the bill small regardless of model

The model choice matters less than what you send it. In order of effect:

1. Send only the relevant skill, never the whole set, and never the SOPs. The skill is a few hundred words; the SOP library is tens of thousands.
2. Send data, not dumps. Resolve the customer and part server-side and pass three fields, rather than pasting a search result with forty columns.
3. Cache the system prompt. The skill and the instructions are identical on every turn and should be billed as such.
4. Summarise the conversation after a handful of turns; an invoice request needs no memory of what was discussed an hour ago.
5. Cap it. A per-department monthly ceiling with a defined behaviour when hit, exactly like the distributor quota.

Prices move, so the plan names a rank rather than a number: Haiku cheapest, then Sonnet, then Opus, then the top tier. Check the current rate card before setting the ceiling, and record the real per-invoice cost from the log after a month — that number, not an estimate, is what decides whether this is worth running.

The local option, honestly

A model running on the Plex server would keep Finance data in the building entirely. On that machine, with no GPU, it would be slow and materially less reliable at exactly the extraction task this depends on. Worth revisiting only if the data sensitivity ever outweighs that, or if the machine gains a GPU.