The agents that change a store keep three jobs apart: deciding what to do, checking whether it may be done, and doing it. Each job is separate code with its own record, so when an outcome is wrong you can see which stage let it through. The planner can be broad and creative because it cannot write anything. The checking stage is narrower, and the executor applies only what got past it.
The three roles
- Planner — reads a bundle of signals for one product or one customer thread and proposes actions as structured tool calls, each with its arguments, its reasoning and a confidence between 0 and 1. It has no route to Shopify, the mailbox or an ad account.
- Judge — checks every proposal before anything runs. Hard constraints are code the model cannot override; a drafted customer reply is also scored by a second, separate model call. Each proposal ends up blocked, queued for a person, or cleared to run.
- Executor — a separate job that picks up cleared work, applies it in the real system, and records the outcome on the same row.
What the judge checks
For the Catalog Agent the judge is a policy layer in code. A price change larger than 30% in either direction is blocked, and its sign must agree with its stated direction. A product with more than 50 units in stock cannot be archived, and the variant tool cannot draft every active variant of a product. A proposal below 0.85 confidence goes to the review queue instead of running.
For the helpdesk, a reply the planner drafts goes to a Reply Judge: a second model, smaller than the planner by default, that scores policy grounding, factual grounding, brand voice, tone and an overall score, and returns pass, edit_required or fail. Deterministic guards then run over the same draft. The policy-grounding score is capped by how many of the reply's policy claims match the store's own policy text. An order or tracking number that appears nowhere in the thread's data, a promise of follow-up work, or a statement that a refund has been processed is caught by its own check. When a person reviews drafts, only pass is sent automatically; when the store has no reviewer, edit_required may also send. fail never sends. If the judge fails to run, the result is marked degraded and is never sent automatically.
{
"verdict": "edit_required",
"policy_grounded": 0.4,
"factual_grounded": 0.9,
"brand_voice_cosine": 0.82,
"tone_match": 0.8,
"overall": 0.66,
"rationale": "Return window stated as 30 days; policy text says 14."
}How a decision is recorded
Catalog proposals are written to the decision log, blocked ones included, with the policy outcome in their evidence: whether it will execute, whether it was blocked and why, and whether a person must review it. The executor then moves each row to applied, skipped or failed. Helpdesk proposals produce one evaluation row per proposed skill, carrying the judge's scores and verdict, plus a draft when a reply was written.
Stopping and undoing
The catalog executor checks the store's kill switch before it applies anything, and a per-store rate limit skips and counts whatever exceeds it. On the helpdesk, the kill switch blocks every skill except defer and escalate. An applied catalog decision can be reverted from the dashboard: the revert applies the inverse Shopify change, such as republishing a drafted product, restoring the previous copy or restoring each variant's original price, and marks the row reverted. A sent email or an issued refund is an event and cannot be taken back, so refunds carry a marker that lets a retry recognise a refund already made instead of paying it twice.
Model calls at the planner and judge stages are charged as credits, derived from what each call actually cost. The deterministic checks make no model call.