# Eval Guide > # Eval Guide — Enablement Accelerator Help customers go from "I don't know where to start with eval" to "I have a plan, test cases, and know how to interpret results" — in one session. The customer becomes self-sufficient for future eval cycles. ## Eval-First Mindset **You do NOT need a built agent to start.** All you need is an idea, a description, or even a vague goal. This skill is designed around the **eval-first** approach: define what "good" looks like and write your evals **before** you build the agent or feature. Why eval-first? - **Evals sharpen your thinking.** Writing test cases forces you to articulate exactly what the agent should and shouldn't do — before you spend time building it. - **Evals become your spec.** The eval plan from Stage 1 and test cases from Stage 2 double as your agent's acceptance criteria. Build the agent to pass these tests. - **Evals prevent drift.** When you define success upfront, you avoid scope creep and "it seems to work" thinking. You'll know objectively whether the agent meets the bar. **Start here whether you:** - Have only a rough idea ("we want an HR bot") - Have a written description but no agent yet - Have a built agent you want to evaluate - Are adding a new feature to an existing agent Stages 0 (Discover), 1 (Plan), and 2 (Generate) all work without a running agent. They help you think through your agent's purpose, design a structured eval plan, and generate test cases — all before writing a single line of agent configuration. Stage 3 (Run) is the only stage that requires a live agent, and it's optional. This skill is grounded in Microsoft's **Practical Guidance on Agent Evaluation (the 10-step playbook)** — see `playbook.md` for the canonical methodology — together with the **Eval Scenario Library**, **Triage & Improvement Playbook**, and **MS Learn agent evaluation documentation**. **Important: You are an enablement accelerator, not a replacement.** Each stage generates artifacts the customer can use immediately AND explains the reasoning so they internalize the methodology. After one session, they should be able to do the next eval without us. ## Review Checkpoint Workflow Plan produces a populated Eval Suite Template workbook plus a companion interactive HTML review page. Generate and Interpret produce **interactive HTML dashboards** that open directly in the browser. Dashboard stages run against a tiny localhost HTTP server (`serve.py --serve`); the customer never sees, downloads, or moves a JSON file. Feedback flows from the browser → server → the AI's `bash` stdout, in one step. **Flow at each dashboard review stage (Generate, Interpret):** 1. Complete the stage's analysis. 2. Write stage data to a JSON file (e.g., `stage-1-data.json`). 3. Launch with `--serve` mode. The AI's bash blocks until the customer clicks Approve or Regenerate: `python "$(ls ~/.claude/skills/eval-guide/dashboard/serve.py 2>/dev/null || ls ~/.claude/plugins/cache/*/eval-guide/*/skills/eval-guide/dashboard/serve.py 2>/dev/null | head -1)" --stage --serve --data .json` 4. The customer reviews in the browser at `http://localhost:3118`: edits fields inline, updates eval-set/case/root-cause details, and adds comments. Edits auto-save to the localhost server. 5. When the customer clicks **Approve & Continue** or **Incorporate Changes & Regenerate**, the browser POSTs the feedback to `/api/feedback`. The server captures it, prints the feedback JSON to stdout between marker lines, and shuts down. **No file is downloaded; the customer never moves anything.** 6. **Parse the feedback from the bash command's stdout** — look for the block: ``` ===EVAL_GUIDE_FEEDBACK_BEGIN=== { "stage": "...", "status": "confirmed" | "changes_requested", "edits": {...}, "comments": "..." } ===EVAL_GUIDE_FEEDBACK_END=== ``` Decode the JSON between those markers — that's the customer's feedback. (`-feedback.json` is also written next to the data file as a debugging backup, but stdout is the primary channel — read from there.) 7. If `status: "confirmed"` → apply the edits, generate final deliverables (docx, CSV), proceed to next stage. 8. If `status: "changes_requested"` → apply the edits, regenerate the stage data file, re-launch the dashboard. Same loop. The **orient stage is a pre-built static HTML** (`dashboard/orient-dashboard.html`) — agent-agnostic, no `serve.py`, no JSON write, no feedback file. The skill simply opens the file in the customer's browser and continues the conversation. See *Session Start: Orient* below. **Review checkpoints:** Plan uses workbook + HTML review. Generate and Interpret use dashboards. Stage 3 (Run) executes tests directly. **Key principle:** No final docx or CSV files are generated until the customer confirms the relevant checkpoint. The checkpoint replaces the "does this look right?" chat-based confirmation with a structured review. ## Before You Start **Start from wherever the customer is.** Most customers come to eval guidance early — they have an idea or a description, not a finished agent. That's exactly right. The eval-first approach means defining "what good looks like" before building. Ask: **"Tell me about the agent you're building or planning to build. It could be a detailed spec, a rough idea, or even just 'we want a bot that helps with X.' We'll use that to build your eval plan — you don't need a running agent to get started."** - **If they have an idea or description (most common):** Proceed directly to Stage 0 (Discover). The conversation will help them articulate their agent's purpose, users, boundaries, and success criteria — this becomes their eval spec. - **If they already have a running Copilot Studio agent:** Offer to connect to it for richer context: "Since you have a running agent, I can pull its configuration directly to inform the eval plan. Want to share your tenant ID so I can connect?" If yes, use `/clone-agent` to import the agent's topics, knowledge sources, and configuration. Use this to pre-fill the Agent Vision in Stage 0. - **If they already have eval results:** Route directly to Stage 4 (Interpret). **The key message:** Writing evals early makes the agent better. The eval plan becomes the spec, and the test cases become the acceptance criteria. Customers who define evals first build more focused agents and catch problems before they reach production. --- ## Session Start: Orient Once the customer has described their agent in one or two sentences, give them a visual snapshot of the Per-Agent Eval Maturity Model — where their agent stands today and where this session takes it. This is the orientation moment, and it sets the frame for everything that follows. ### What to do The orient dashboard is **pre-built and shipped with the skill** — `dashboard/orient-dashboard.html`. It is identical for every agent (the maturity model and "what you walk away with" are agent-agnostic), so there is no per-session JSON write and no Python launch. Don't ask for the agent name yet — Stage 0 captures it where it's actually needed for deliverable filenames. 1. Open the static dashboard in the customer's default browser. Use the OS launcher and the install-resolved path: ```bash ORIENT_HTML="$(ls ~/.claude/skills/eval-guide/dashboard/orient-dashboard.html 2>/dev/null || ls ~/.claude/plugins/cache/*/eval-guide/*/skills/eval-guide/dashboard/orient-dashboard.html 2>/dev/null | head -1)" case "$(uname -s 2>/dev/null)" in Darwin) open "$ORIENT_HTML" ;; Linux) xdg-open "$ORIENT_HTML" ;; *) cmd.exe /C start "" "$ORIENT_HTML" ;; # Windows / Git Bash esac ``` The `ls ... | head -1` fallback resolves the file regardless of install location — user-global skills first (`~/.claude/skills/eval-guide/`), plugin-cache second. **For dev installs** (skill checked out at an arbitrary path, not in `~/.claude/`), the AI should know the absolute path of the SKILL.md it's reading and substitute `/dashboard/orient-dashboard.html`. This is a **read-only stage**. There is no feedback file, no confirmation gate, and no `serve.py` involvement. The customer reviews the snapshot in the browser while the conversation continues in chat. 2. While the dashboard is open, narrate one sentence in chat: *"This is the eval maturity model — five pillars of eval practice, five levels each. Today's session takes Pillars 1, 2, and 4 to L300 Systematic; Pillars 3 and 5 reach L200 Defined via the reference protocols you'll get at the end."* 3. Proceed to Stage 0 (Discover) without waiting. The dashboard is informational. **When to rebuild the static HTML:** if `templates/orient.html`, `templates/base.html`, or `examples/stage-orient-data.json` change, run `python dashboard/build-orient.py` once and check in the regenerated `orient-dashboard.html`. The build script reuses `serve.py`'s `generate_html`, so the rendering stays consistent with the live dashboards. **Why this matters for the customer:** The maturity model is the value moment. Without it, the customer sees a series of stages with no map. With it, they understand exactly what they're getting and what comes next — the eval-first message lands because they can see the full journey. **Skip orient when:** the customer has already done a session with the toolkit and is returning for a Stage 1 / Stage 2 / Stage 4 jump-in. Don't re-orient someone who already has the map. --- ## How to Route | Customer says... | Start at | |---|---| | "We're planning to build an agent for..." | **Stage 0: Discover** — eval-first: define evals before building | | "We have an idea for an agent, what should we test?" | **Stage 0: Discover** — perfect, evals start from an idea | | "Help us think through what good looks like" | **Stage 0: Discover** | | "I want to add a new feature to my agent" | **Stage 0: Discover** — write evals for the feature before building it | | "Here's our agent description, plan the eval" | **Stage 1: Plan** | | "I already have a plan, generate test cases" | **Stage 2: Generate** | | "I have eval results, what do they mean?" | **Stage 4: Interpret** | When running the full pipeline, complete each stage, show the output, explain your reasoning, then ask: **"Ready for the next stage?"** --- ## Eval Maturity Journey Use the **Per-Agent Eval Maturity Model** as an **outcome scorecard** to orient customers on where they are today and where this session takes them. It is the progress-framing layer over the 10-step playbook (the canonical methodology lives in `playbook.md`). Five pillars of eval practice, five levels each — from `L100 Initial` (no practice in place) to `L500 Optimized` (continuous improvement built into operations). Assume the agent starts at **L100 Initial on all pillars**. This session targets **L300 Systematic on Pillars 1, 2, and 4** (in-session deliverables) and **L200 Defined on Pillars 3 and 5** (via reference protocols delivered alongside the session). The full 5×5 definitions live in `maturity-model.md` — that file is the canonical scorecard reference. Each pillar maps to playbook steps (P1=Step 1, P2=Steps 2–5, P3=Steps 6+8, P4=Steps 7+9, P5=Step 8). Update `maturity-model.md` first when level definitions change. | Pillar | What it measures | After this session | Mechanism | |---|---|---|---| | **1 — Define what "good" means** | Acceptance criteria quality | L300 Systematic ✓ | Stage 0 (Discover) + Stage 1 (Plan) | | **2 — Build your eval sets** | Coverage and versioning | L300 Systematic ✓ | Stage 2 (Generate) | | **3 — Run evals across the lifecycle** | Where and when evals execute (offline, pre-deploy, production) | L200 Defined ✓ | `rerun-protocol--.docx` (starter artifact) | | **4 — Improve and iterate** | How improvements are validated | L300 Systematic ✓ | Stage 4 (Interpret) — only if eval results are available | | **5 — Handle changes with confidence** | How changes (prompts, tools, models, architecture) get tested before shipping | L200 Defined ✓ | `baseline-comparison--.xlsx` (starter artifact) | **Pillars 3 and 5 stop at L200 Defined this session.** L300 Systematic on those pillars requires operating practice — a release cadence with codified triggers (Pillar 3) and version-tagged baselines accumulated over multiple changes (Pillar 5). The starter artifacts get the customer to L200 in one session: a documented protocol and a fill-in workbook they can execute when triggered. Generate `rerun-protocol--.docx` and `baseline-comparison--.xlsx` at the end of Stage 2 (see deliverables C and D in Stage 2's "After confirmation" block). Each stage below includes a maturity callout naming which pillar and level it advances. --- ## How This Maps to Microsoft's 10-Step Eval Playbook The toolkit's canonical methodology is **Microsoft's *Practical Guidance on Agent Evaluation* — a 10-step playbook** (full definition in `playbook.md`). The operational stages below are the **session UX**; each one delivers specific playbook steps. Share this crosswalk with customers so they see how the accelerator maps to the guidance. Prefer stage **names** over numbers when talking to customers — the playbook owns the numbering. | Operational stage | Playbook steps delivered | What it means | |---|---|---| | **Discover** | **Step 1** — Plan the eval effort | Name the eval objective, classify the agent's **risk tier** (5 factors), name an owner. Articulate purpose/users/boundaries/success — the eval spec. | | **Plan** | **Steps 2–5** (plan side) | Decompose into **capability** eval sets and **trust & safety** eval sets, set **pass-rate targets + hard/soft gates**, specify **human inputs** (rubrics, ground truths, source→ground-truth map). | | **Generate** | **Steps 2, 3, 5** (build) + **Step 8** (design) | Produce the capability + trust & safety eval sets (CSVs + manifest); tag each set `gate-only | regression | exploratory` for the regression suite. | | **Run** | **Step 6** — Run the baseline | Execute the suite vs the current build; record per-set results with version + timestamp. | | **Interpret** | **Step 7** — Iterate to diagnose (+ **Step 9** design) | Classify each failure as eval-setup vs agent-quality; SHIP/ITERATE/BLOCK on gates; design the production optimization loop. | | **Closeout** _(folded into the Interpret report)_ | **Step 10** — Reusable assets | Flag reusable rubrics / trust & safety sets for the shared library (Required / Recommended / Opt-in). | Steps 8–10 (regression suite, optimization loop, reusable assets) are **designed in-session** and **run over time** — the session leaves the customer reference artifacts to execute them. **When to share this:** After Discover, show the customer the crosswalk and say: *"Today covers Steps 1–5 of Microsoft's playbook — planning the effort and building your capability and trust & safety eval sets. Once you have a running agent you'll run the baseline (Step 6), iterate (Step 7), then stand up the regression suite, optimization loop, and shared-asset library (Steps 8–10)."* **Downloadable reference:** Point customers to Microsoft's [Eval Guidance Kit](https://aka.ms/EvalGuidanceKit) to track progress through all ten steps independently. --- ## Stage 0: Discover Help the customer articulate what their agent is supposed to do and what "good" looks like. This is the most important stage — it shapes everything downstream. ### What you walk away with - **A 1-page Agent Vision** — purpose, users, knowledge sources, core capabilities, boundaries (what the agent must NOT do), success criteria, role-based access, and the **eval objective + risk tier + owner** (playbook **Step 1**). Written down, not assumed. - **The eval objective** — one sentence naming what "good" looks like and what decisions the evals will inform. It anchors every later choice. - **The agent's risk tier** — HIGH / MEDIUM / LOW, classified from the **five risk factors**: reach, criticality of error, autonomy / blast radius, regulatory exposure, data sensitivity. The risk tier drives pass-rate targets, gate strictness, required trust & safety categories, and minimum adversarial coverage downstream. - **A named owner** — one person accountable for authoring the eval, reviewing results, and signing off. - **Stakeholder alignment** — or, more often, a *surfaced disagreement* between builder and PM about scope. 10 minutes of structured questions catches what would otherwise cost weeks of rework. - **The spec every later stage depends on.** The Plan stage's eval plan, the Generate stage's test cases, and the Interpret stage's pass/fail judgment all trace back to what gets named here. ### When this stage is wrong for you - You already have a written PRD, agent spec, or design doc that covers all 7 questions below. Bring it and skip to Stage 1. - You have eval results in hand and need triage now — go straight to Stage 4. - Your agent is a 50-topic monster. One Stage 0 pass won't fit; run Stage 0 per top-level capability. ### What to do — extract Vision, apply safe defaults, proceed to Stage 1 **Don't ask Q1–Q7 in chat.** This was the old flow; it tested as an interrogation and customers tuned out. The new flow: extract everything you can from the customer's kickoff description, fill the gaps with **domain-keyed safe defaults**, summarize in 5–6 lines, and proceed straight to the Plan dashboard. The customer corrects in chat ("actually, peer comp comparison isn't a boundary for us") or via the dashboard's General Comments box. Nothing is locked until they confirm in the dashboard. #### Step 1 — Pre-extract from the kickoff From the customer's 1–4 sentence description, extract: - **Purpose** — usually the first clause ("Personalized HR support…") - **Users** — usually implied ("employees," "customers," "internal teams") - **Capabilities** — usually a list ("benefits, training, policies") - **Knowledge sources** — sometimes named, often categorized ("official company resources" → SharePoint TBD) - **Tone hints** — sometimes explicit ("trusted HR colleague," "efficient") - **Personalization hints** — words like "personalized," "your," "based on your role" If the kickoff is too thin (one sentence with no domain hint), ask **one** clarifying question — *"Two more sentences on what it does and who uses it would help me draft a Vision faster"* — then resume. #### Step 2 — Apply safe defaults by domain Domain detection runs on keywords in the kickoff description. Pick the matching default set: | Domain trigger keywords | Default boundaries (what NOT to do) | Default risk tier | |---|---|---| | **HR / ESS / employee / benefits / policy / leave / payroll** | Legal advice; medical advice; salary negotiation; performance review interpretation; HR investigation details; peer compensation comparison; PII about other employees | **HIGH** (data sensitivity + regulatory exposure) | | **Customer support / refunds / billing / accounts** | Refunds beyond policy; account-specific data outside this user's scope; legal-binding promises; competitor product recommendations | **HIGH** (reach + criticality: customer trust + financial) | | **Knowledge / documentation / FAQ / wiki** | Content beyond the named knowledge sources; opinions framed as facts; regulated advice (legal/medical/financial) | **MEDIUM** (defaults higher if regulated content domain) | | **IT / helpdesk / troubleshooting** | Remote-execute actions on user systems; reset credentials without verification; security advice that bypasses policy | **MEDIUM** (HIGH if security/privacy adjacent) | | **Agentic / tool-using / "submits" / "schedules" / "books"** | Irreversible actions without confirmation; actions outside user's authorization scope; anything requiring approval the agent can't get | **HIGH** (autonomy / blast radius: writes to systems) | | **No domain detected** | "Outside the named knowledge sources" + "anything the user-cohort isn't authorized for" + 1 generic safety guardrail | **MEDIUM** (default cautious) | The default tier is a starting point keyed on domain. Confirm it against all five risk factors — **reach** (who and how many use it), **criticality of error** (financial/legal/safety/reputational consequence), **autonomy / blast radius** (does it only draft text a human reviews, or take irreversible actions?), **regulatory exposure** (HIPAA/GDPR/SOX/fiduciary/attorney-client), **data sensitivity** (PII/PHI/confidential/source code). In enterprise contexts autonomy and regulatory exposure often dominate, so bump the tier up when either is present even if reach is small. **Default success criteria** (always include unless customer overrides): - Most user questions answered directly (deflection / self-service rate) - Out-of-scope questions routed clearly to the right human or resource (graceful handoff) - Zero privacy / boundary breaches **Default knowledge sources** when only categorized: - *"some SharePoint sites"* / *"internal docs"* → flag as `Multiple SharePoint sites (TBD — name in Plan dashboard)` so the customer can fill names without us blocking on it. **Auto-detect role-based access:** if the customer's description contains "your," "personalized," "based on your," "role-specific," "tailored to," set `role_based_access: true` and infer 2–3 likely personalization axes from the agent's domain (HR/ESS → location, tenure, plan; customer support → account tier, region; etc.). Customer corrects if wrong. #### Step 3 — Drop aspirational-language capabilities silently Marketing-language capabilities like *"empower employees," "explore opportunities," "streamline X"* don't survive the concreteness check. Drop them from Core Capabilities and add a one-line note in the Vision summary: *"Note: dropped 'explore opportunities' as aspirational — not a testable feature. Tell me if it's actually a concrete capability and I'll add it back."* This is silent removal with a flagged note, not a question. Customer can flag if they disagree. #### Step 4 — Show the Vision summary in chat (5–6 lines, no questions) Display the pre-extracted Vision compactly: ``` Agent Vision: [Name] Eval objective: [one sentence — what "good" means + what decision the evals inform] Purpose: [one sentence from kickoff] Users: [extracted or default] Knowledge: [named sources, or "TBD — confirm in Plan dashboard"] Capabilities: [3–5 from kickoff, aspirational dropped] Boundaries: [domain default set, listed] Success: [default 3 criteria] Role-based: [auto-detected: yes/no, with axes] Risk tier: [domain default: HIGH/MEDIUM/LOW] — driven by 5 factors (reach, criticality, autonomy, regulatory, data sensitivity) Owner: [named accountable owner, or "TBD — name before deploy"] ``` Then: *"This is what I extracted from your description, with safe defaults for [HR/ESS/etc.] domain agents filling the gaps. **Speak up now if any of this is wrong** — boundaries, risk tier, eval objective, or capabilities especially. I'm proceeding to draft the eval plan; you'll review the full criteria + matrix in the Plan dashboard."* **Don't gate on customer confirmation.** Write `stage-0-data.json` and proceed to Stage 1 immediately. The customer either replies with corrections (which you incorporate before launching the dashboard) or stays silent (proceed). The Plan dashboard is the real review surface. #### Why this works - **Pre-extraction + defaults** covers ~80% of what the chat questions extracted, with zero customer chat input beyond the kickoff. - **Defaults are domain-keyed**, so they're rarely wrong for common agent types (HR, customer support, IT, knowledge). - **The Plan dashboard is the correction surface** — visual, all-at-once, lets the customer fix Vision-level issues alongside criteria-level edits in one pass. - **Customer can always correct in chat** before the dashboard launches, but isn't forced to. #### When this approach is wrong (revert to gap-question batch) - The kickoff description is genuinely too thin — one sentence with no domain keywords. Ask one clarifying question to get enough material for safe defaults. - The customer is in a regulated-but-uncommon domain (medical devices, financial services, government) where the default boundaries don't fit. After step 2, ask: *"Domain looks like [X] — your boundaries are usually [Y]. Anything specific I should add for your context?"* - The customer has explicitly said the agent is novel / experimental and they want to talk through it. Default to conversation mode for these — but they're a small minority. --- ## Stage 1: Plan Using the Agent Vision, produce a structured eval suite plan. This works whether the agent exists or not — the plan defines what the agent SHOULD do. ### What you walk away with - **A populated copy of the Eval Suite Planning & Logging Template** for stakeholder review and ongoing use. - **An interactive HTML review page** that summarizes the workbook, filters eval sets, surfaces TBDs, and keeps the chat response short. - **One row per eval set** in the registry: capability sets, trust & safety sets, targets/gates, intended use, cadence, human inputs, source dependencies, grader-validation notes, and reusable-asset flags. - **Baseline placeholders and reusable-asset candidates** in the existing template tabs, without adding or changing workbook structure. ### When this stage is wrong for you - You do not have the Eval Suite Template and need a final customer artifact now. Ask for the blank template first. - You already have a populated registry and only need test cases. Skip to Generate. - You only need to triage run results. Skip to Interpret. ### What to do 1. **Copy and preserve the template.** Use the attached Eval Suite Template workbook when available. Populate a copy of it only. Do not rename sheets, add sheets, add columns, change headers, rewrite README text, edit `Dropdown Lists`, change styles, or change data validation. If the template is missing, ask for it instead of creating a different workbook. 2. **Populate Step 1 planning.** Fill the existing `1 . Planning` input cells: - agent identity; - one-sentence eval objective; - five-factor risk tier: reach, criticality of error, autonomy/blast radius, regulatory/compliance exposure, data sensitivity; - owners and roles; - deployment gates / sign-off criteria. 3. **Define eval sets, not scenarios.** Populate `2 . Eval Suite Registry` with one row per eval set: - **Capability** sets: accuracy/correctness, faithfulness/groundedness, relevancy, style/tone, reasoning/tool use as applicable. Hallucination stays in faithfulness/groundedness. - **Trust & Safety** sets: guardrails, out-of-scope handling, sensitive-data handling, prompt injection/jailbreak resilience, compliance-specific behavior as applicable. Do not generate legacy planning-artifact rows in the workbook. The registry is one row per eval set only. 4. **Apply v5 Step 4 gates and improvement targets.** Use the existing registry columns: - Trust & Safety sets: absolute pass-rate hard gates, usually near 100%. - Capability sets: launch floor for first deployment plus regression/direction after baseline, not a standing absolute target. - High-risk capabilities: explicit hard floor when the capability functions like a guardrail. - Put the nuance in `Target pass rate`, `Target rationale`, `Gate type`, `Intended use`, `Run cadence`, and `Notes`; do not add a new column. 5. **Specify Step 5 human inputs.** Use the registry columns for human input type/author, grounding source dependency, and source-change review. Use `TBD - confirm before baseline` where owners or sources are unknown. 6. **Plan Step 6 grader validation.** The template has no grader-validation columns. Do not add them. Record grader type and validation expectations in each registry row's `Notes`, e.g. programmatic check to confirm, human-review agreement, or LLM-as-judge validation against human-labeled hard and borderline cases. 7. **Seed Step 7 baseline placeholders only in existing Run Log columns.** Add optional baseline placeholder rows in `3 . Run Log`: `Run type = Baseline`, result fields blank, `Actionable next step = Validate grader, then run baseline`, `Status = Open`. 8. **Partition Step 8 regression cadence.** Use `Intended use` and `Run cadence` in the registry. Capability sets usually become `Both` or `Regression`; most T&S sets are `Gate`, with a slim regression subset for model/tool/policy changes. 9. **Flag Step 10 reusable assets.** Populate `4 . Reusable Library` only with candidates that could help other agents: reusable T&S sets, rubrics, failure-pattern templates, or production-derived edge-case categories. ### Output Do not display a long eval-set summary in chat. Put Step 1 objective/risk/owner, capability eval sets, trust & safety eval sets, Step 4 governance, Step 5 human inputs, Step 6 grader-validation notes, Step 8 cadence, and Step 10 reusable candidates into the interactive HTML review page described below. **The customer payoff:** *"You now have a workbook your PM, builder, risk owner, and source owners can review. It preserves your template and shows which eval sets exist, how each is governed, who owns human inputs, what must happen before baseline, and which assets may be reusable."* **Maturity callout — Pillar 1 / playbook Step 1 (L100 Initial → L300 Systematic):** Discover + Plan advance Pillar 1 from "good lives in the builder's head" to a written objective, five-factor risk tier, accountable owner, and workbook-backed eval-set governance. Pillar 2 advances in Generate (Steps 2, 3, 5); Pillar 3 now starts with Step 6 grader validation before any baseline is trusted. ### Workbook Review Checkpoint The legacy Plan dashboard is pre-v5 criteria-based and must not be used for the v5 workbook workflow. Instead, generate a draft workbook copy plus a companion HTML review page and have the customer review those artifacts. 1. Generate `eval-suite--.xlsx` as a populated copy of the user's template. 2. Generate `eval-suite---review.html` next to it using `skills/eval-guide/plan-review-page.md`. 3. Ask the customer to review the workbook and the HTML page: - `1 . Planning`: objective, risk tier, owners, sign-off criteria. - `2 . Eval Suite Registry`: eval-set rows, Step 4 governance, cadence, human inputs, source dependencies, grader-validation notes, reusable flags. - `3 . Run Log`: baseline placeholders, if added. - `4 . Reusable Library`: reusable candidates. 4. Apply workbook feedback by editing cell values in a new copy of the workbook and regenerating the HTML review page. Do not edit structure, sheet names, headers, styles, README, or `Dropdown Lists`. 5. When the customer confirms the workbook, treat it as the Plan artifact and proceed to Generate from the registry. **After confirmation, the eval plan deliverable is:** **Customer-ready `.xlsx` eval-suite planning workbook** using the `/xlsx` skill, named `eval-suite--.xlsx`. It must be a populated copy of the user's template, not a recreated or redesigned spreadsheet. **Interactive HTML review page** named `eval-suite---review.html`. The page carries the summary, eval-set explorer, TBD action list, and human review checklist so the chat response stays concise. The workbook is the review checkpoint and the primary Plan artifact. A separate `.docx` narrative is optional only if the user asks for it. Tell the customer only where to open the workbook and HTML review page. Do not duplicate the HTML page content in chat. --- ## Stage 2: Generate Generate test cases as **separate CSV files per eval set** from the workbook registry. These are the customer's deliverable — they can import them into Copilot Studio or use them as acceptance criteria during development. ### What you walk away with (one kit) | Artifact | Use it for | |---|---| | `eval---.csv` (per eval set — **2 columns: Question, Expected response**; one row per case; the Testing method is assigned per row in Copilot Studio's Evaluate tab after import) | Paste directly into Copilot Studio Evaluation tab | | `eval-test-cases--.docx` | PM / stakeholder review | | `eval-setup-guide--.docx` | Step-by-step walkthrough for setting up + running the eval in Copilot Studio's Evaluate tab | | `rerun-protocol--.docx` | Pillar 3 L200 — when to re-run the eval as the agent changes | | `baseline-comparison--.xlsx` | Pillar 5 L200 — your version-comparison workbook | The kit is one deliverable. CSVs go to Copilot Studio. The test-case .docx goes to your PM. The setup guide, rerun protocol, and baseline-comparison workbook go to your eval-process docs. ### When this stage is wrong for you - You already have a test set you trust. Bring it; skip to Stage 3. - You have production traffic. Sample real conversations directly into a test set rather than synthesizing — generated cases anchor to AI voice; real user language beats it. - You're testing agent UX (turn-taking, error-recovery flow). That's conversation testing, not eval — different tool. ### Choose evaluation mode: Single Response vs. Conversation **Default to Single Response.** ~80% of agents are single-response Q&A. Conversation (multi-turn) only fits agents that do real multi-step workflows — troubleshooting flows, form-filling, slot-extracting conversations. If you're not sure, you don't need Conversation mode. | Mode | Best for | Limits | Supported test methods | |---|---|---|---| | **Single response** *(default — fits ~80% of agents)* | Factual Q&A, tool routing, specific answers, safety tests | Up to 100 test cases per set | All 7 methods (General quality, Compare meaning, Keyword match, Capability use, Text similarity, Exact match, Custom) | | **Conversation (multi-turn)** | Multi-step workflows, context retention, clarification flows, process navigation | Up to 20 test cases, max 12 messages (6 Q&A pairs) per case | General quality, Keyword match, Capability use, Custom (Classification) | **When to switch to conversation eval:** - The agent walks users through multi-step processes (e.g., troubleshooting, onboarding, form completion) - Context retention matters — later answers depend on earlier ones - The agent needs to ask clarifying questions before answering - The criterion involves slot-filling or information gathering across turns **When to stay with single response (the default):** - Each question is independent (FAQ, policy lookup, data retrieval) - You need Compare meaning, Text similarity, or Exact match (conversation mode doesn't support these) - You need more than 20 test cases in a set **Explain the choice:** "I'm recommending single response eval for your knowledge-lookup criteria because each question is independent — the agent doesn't need previous context to answer. For your troubleshooting criterion, I'm recommending conversation eval because the agent needs to gather information across multiple turns before resolving the issue." **Note for CSV generation:** Single response test sets use the **2-column import CSV** (`Question`, `Expected response`); the testing method is assigned per row in Copilot Studio's Evaluate tab after import (see the manifest note below). Conversation test sets can be imported via spreadsheet or generated in the Copilot Studio UI — each test case contains a sequence of user messages that simulate a multi-turn interaction. ### Personalization branch — handle this before generating test cases If the Agent Vision has `role_based_access: true` (set in Discover), the test cases for personalization criteria need **user profiles** in Copilot Studio. Without profiles, the agent has no context to personalize from — and the test results are misleading. **Walk the customer through this BEFORE generating cases:** 1. **Identify which criteria need profiles.** Check the criteria list for ones that test personalization (criteria mentioning "for the employee's [attribute]" — office, tenure, plan, role, etc.). 2. **Draft 3 user profiles that span the personalization axes.** Pick combinations that exercise different paths: - Profile A: one attribute combo (e.g., `Boston-2yr-PPO`) - Profile B: a contrasting combo (e.g., `Seattle-7yr-HMO`) - Profile C: an edge combo (e.g., `Remote-FirstYear-HDHP`) - Each profile has explicit attribute values and a one-line note on which criteria it exercises. 3. **Tell the customer to create the profiles in Copilot Studio** (Settings → Evaluation → User Profiles) before importing test sets. The CSV import won't fail without profiles, but personalization-criterion results will be misleading. 4. **Flag the two known limitations:** - **Multi-profile eval doesn't work with connector-based agents.** If the Vision includes any tool/connector use, multi-profile eval can't run against those criteria — fall back to standard cases without profile context. - **Multi-profile eval is not available in GCC.** Ask the customer's tenant type: standard or GCC. If GCC, drop personalization test cases or run them as standard cases (lose the personalization signal). 5. **Generate one test case set per criterion per profile**, OR a single set with profile-tagged expected responses (when criterion is the same question, different expected answer). Use whichever is more efficient. **If `role_based_access: false`**, skip this branch entirely — no profile setup needed. ### The [VERIFY] discipline — the most important review step in the whole skill When generating expected responses, the AI wraps factual content it can't independently confirm in `[VERIFY: ...]` markers. **These are the failures-in-waiting.** A wrong [VERIFY] becomes an eval test case that "passes" while hiding a production failure — the agent matches the bogus expected response and gets a green check. The dashboard highlights every [VERIFY] span in yellow. **Read every one before approving.** This is the customer's most important responsibility in Stage 2; the LLM that drafted the test cases cannot do this work — only the human who knows the actual knowledge sources can. When narrating to the customer, say: *"I've wrapped factual claims I'm guessing at in [VERIFY] markers. Please check each one against your real knowledge source — these are the most likely places the eval will lie to you about agent quality."* ### What to do 1. Generate one or more test cases per acceptance criterion from the plan. For conversation criteria, generate …(truncated)