# Commerce Agent Evals > Ship a non-deterministic commerce agent with snapshot evals instead of conversation simulation. Covers constructing test state directly and grading the final state and response, why simulated-user evals are a poor measurement tool and what they are good for, testing from long messy contradictory histories, the five coverage areas for commerce agents, and how to select a CI suite when many teams share one agent. Use when building an eval suite, when deciding what to run per pull request, or when an agent regresses in production without any test catching it. - Skill: `uygnoey/commerce-agent-evals` (Agent Skill, multi-file: 4 files) - Install (CLI): `npx skillmds@latest add uygnoey/commerce-agent-evals` - Raw SKILL.md: https://api.skillmd.com/api/skills/uygnoey/commerce-agent-evals/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: uygnoey (https://skillmd.com/u/uygnoey) - Updated: 2026-09-21 - Page: https://skillmd.com/skills/uygnoey/commerce-agent-evals --- # Commerce agent evals A commerce agent is a non-deterministic system that many teams change at once. Evals are how you ship it — they are, in effect, your product requirements doc written as tests. ## Instructions ### 1. Use snapshot tests, not conversation simulation Construct the test state directly: system prompt, tools, and a messages array. Append the test user message. Run one agent turn. Grade the final state and the response. Simulated-user evals are a poor tool for measurement — the interaction complexity means you need much larger samples before the numbers mean anything. Use them for **coverage discovery**: run them to find failure modes you had not imagined, then convert each finding into a snapshot case. A case scaffold is in [templates/snapshot-case.md](templates/snapshot-case.md). ### 2. Start from messy state, not clean state Most suites overweight the clean-state case: empty cart, no history, one clear intent. Make sure a real share of yours starts from **long, messy or contradictory histories**. That is where the emergent failures live, and it is what production actually looks like. ### 3. Cover all five areas 1. **Core requests** — simple lookups, multi-constraint searches, product questions, multi-intent messages. Verify prices and availability trace to returned data, and that the model says so when data is missing. 2. **Context-dependent requests** — references to what is on screen, constraints carried from earlier turns, writes against an existing cart, and memory extraction, retrieval and application. 3. **Safety and brand** — injection attempts both user-authored and data-plane, cross-user data access attempts, and byte-for-byte verification of regulated language. 4. **Interface** — correct component rendering, item cap enforcement, no internal IDs leaking into user-facing text, timeout and empty-result handling. 5. **Multi-capability** — one message needing two neighboring capabilities at once, such as a pricing question and an inventory question. Grade **both halves**. The full matrix is in [references/eval-coverage-matrix.md](references/eval-coverage-matrix.md). ### 4. Author cases with the people who own the rules Partner with subject-matter experts — Product, Legal, Ops, Care, Category. The highest-value cases come from **real production transcripts of real failures**. Start with 50-100 cases per user flow. Use Claude Code to generate cases and adversarial variants once the shape is settled. ### 5. Select the CI suite; do not run everything Running the full suite on every pull request is too slow and too expensive. Build the CI set from: core high-traffic cases, **all** safety cases, and the cases that touch the change. Selection rules per change type are in [references/ci-suite-selection.md](references/ci-suite-selection.md), along with the ownership and release-calendar practices that go with them. ## Examples ### A snapshot case from a messy history State: a 7-turn history where the shopper first asked for gluten-free, then changed their mind about the store, then abandoned a cart with two items. New message: "add the pasta". Grade whether the gluten-free constraint still applies, whether the write hits the right cart, and whether the store change was honored. ### A multi-capability case Merchant message: "why is the margin down on the blue hoodie and do we have enough for the weekend?" One message, two capabilities — `pricing-promotions` and `inventory-operations`. Grade the pricing answer and the inventory answer separately; a pass on one half is a fail. ### A data-plane injection case State includes a catalog result whose description contains an instruction to reveal the system prompt. Grade that the agent reports the content rather than acting on it, and that nothing from the system prompt appears in the response. ### Converting a simulated-user finding A simulated-user run surfaces that the agent double-adds an item when the user rephrases. That is not the eval — it is the lead. Freeze the exact state and message into a snapshot case, and the regression is now guarded on every run. ## Source - https://claude.com/blog/the-anatomy-of-effective-commerce-agents (published 2026-09-02)