Eval Result Interpreter

Purpose

This skill takes eval results - a Copilot Studio evaluation CSV file (the primary worked example), an export from your own evaluator/harness, a pasted summary, or a plain-English description of results - and produces a structured triage report. It is the final step in the eval lifecycle: plan → generate → run → interpret. The output tells you whether to ship, what broke, why it broke, and what to fix first.

Platform context. All the operational examples below use Microsoft Copilot Studio because its evaluation CSV format is well-documented and concrete. The triage framework itself (4 layers, 3 root cause types, SHIP/ITERATE/BLOCK verdict) is platform-agnostic - point it at any evaluator output with per-case results and the same analysis applies.

This skill serves Stages 2-4 of the MS Learn 4-stage evaluation framework. In Stage 2 (Set Baseline & Iterate), it interprets your first eval results and guides fixes. In Stage 3 (Systematic Expansion), it identifies coverage gaps worth expanding into. In Stage 4 (Operationalize), it triages regression failures after agent updates. Use the evaluation checklist template to track which stage you are in and what to interpret next.

Knowledge source: This skill's analysis framework is grounded in Microsoft's Triage & Improvement Playbook (github.com/microsoft/triage-and-improvement-playbook) - the 4-layer triage system, SHIP/ITERATE/BLOCK decision tree, 3 root cause types, 26 diagnostic questions, and remediation mapping.

When to use this skill vs. eval-triage-and-improvement

These two skills share the same triage framework but serve different modes of work:

If in doubt: Start with eval-result-interpreter to get the structured report, then switch to eval-triage-and-improvement if you need interactive help implementing the fixes.

Instructions

When invoked as /eval-result-interpreter <results>, parse the input and produce the output below. Accept any of these input formats:

Format 1 - Copilot Studio CSV file (primary worked example; use as the template for other platforms)

The user provides a file path to a CSV exported from Copilot Studio agent evaluation. The CSV has these columns:

A single row may have multiple test methods: testMethodType_2, result_2, passingScore_2, explanation_2, etc.

When the user provides a file path, read the CSV and parse it. Count Pass/Fail totals and per test method.

Format 2 - Plain-text summary

A pasted pass/fail count, list of failures, or verbal description of results.

Format 3 - Scenario plan reference (optional, improves accuracy)

If the user also provides the scenario plan table from /eval-suite-planner, use it to map each CSV row to its original category (core business, capability, safety, edge case) and Scenario ID. This is more accurate than inferring categories from question content alone. Say: "Using your scenario plan for category mapping."

Work with whatever detail is available. If input is sparse, state what you assumed. Do not ask for more - give the best triage possible with what is provided.

Output structure

0. Pre-triage infrastructure check (per the Triage Playbook)

Before analyzing failures, verify infrastructure was healthy during the eval run. If any of these were unhealthy, mark affected cases as infrastructure-blocked, not agent-failed:

• Were all knowledge sources accessible and fully indexed?
• Did any API backends return errors, timeouts, or rate-limiting?
• Were authentication tokens valid throughout the run?
• Did the eval environment match the intended configuration?

If you cannot determine infrastructure health from the input, state: "Infrastructure health not verifiable from this input - proceeding with analysis. If failures seem inconsistent, re-run after verifying all knowledge sources and APIs are accessible."

1. Score summary

Parse the results and produce:

If the CSV has multiple test methods per row, also report pass rate per method.

2. Verdict - per the Triage Playbook's SHIP/ITERATE/BLOCK decision tree

Apply this decision tree from the Playbook:

ALL safety/compliance test cases above blocking threshold (>=95%)?
    NO  -> BLOCK: Fix safety issues before anything else.
    YES ->
        ALL core business test cases above threshold (>=80%)?
            NO  -> ITERATE: Focus on the lowest-scoring area.
            YES ->
                Capability test cases above threshold?
                    NO  -> SHIP WITH KNOWN GAPS: Document gaps, monitor.
                    YES -> SHIP.

Use risk-based thresholds (from the Playbook's Layer 1). Adjust for context:

If the CSV does not include tags or categories, infer from the question content whether each case is core business, capability, or safety. State your inference.

State the verdict prominently:

• "Verdict: SHIP." - All signals above thresholds.
• "Verdict: SHIP WITH KNOWN GAPS." - Core passing, some capability gaps documented.
• "Verdict: ITERATE." - Core business or important signals below threshold.
• "Verdict: BLOCK." - Safety failures OR overall pass rate <60%.

If pass rate is 100%: "A 100% pass rate is a red flag - your eval is likely too easy. Add harder edge cases and adversarial scenarios before trusting this result."

3. Failure triage - per the Triage Playbook's Layer 2

For each failing test case (or cluster of similar failures), apply the Playbook's 5-question eval verification sequence FIRST, before blaming the agent:

If the eval passes all 5 checks, classify using the Playbook's 3 root cause types:

• Eval Setup Issue - the test case, expected answer, or test method is wrong. The agent may be performing correctly. Per the Playbook: at least 20% of failures in a new eval are eval setup issues, not agent issues. Sub-types: outdated expected answer, overly rigid grader, unrealistic test case, wrong eval method, grader factual error, grader systematic bias, ambiguous acceptance criteria.
• Agent Configuration Issue - the agent genuinely produced a bad response. Fix via system prompt, knowledge sources, tool config, or topic routing.
• Platform Limitation - caused by underlying platform behavior you cannot fix through configuration. Indicators: same failure persists across multiple prompt/config variations; retrieval consistently returns wrong documents despite correct config. Document and design a workaround.

Group failures that share a root cause. For example: "Cases 3, 5, and 7 all fail with 'Question not answered' - this is likely a single agent configuration issue (missing knowledge source or scope gap), not three independent problems."

3b. Platform diagnostic tools (recommend when applicable)

Copilot Studio provides built-in tools that accelerate triage. Reference these when they would help the customer investigate further:

When triaging failures, always suggest the activity map for cases where you cannot determine root cause from the CSV explanation alone. The activity map is the single most useful diagnostic tool - it shows you exactly what the agent "thought," not just what it said.

Supplementary signal: User reactions (thumbs up/down)

If the agent is already deployed (even in preview), Copilot Studio captures user reactions - thumbs up/down on agent responses. These are not part of the eval CSV, but they complement eval results:

• Eval says PASS but users give thumbs down: The eval may be too lenient, or the test cases may not represent real user expectations. Investigate the gap between what the grader accepts and what users actually want.
• Eval says FAIL but users give thumbs up: The eval may be too strict (grader rigidity), or users have lower standards than the eval. Revisit the expected responses for these scenarios.
• Cluster of thumbs-down on a topic not covered by eval: Coverage gap - add test cases for that topic area in the next eval iteration.

If user reaction data is available, mention it in the pattern analysis (section 6) to cross-reference eval results with real-world satisfaction. Do not treat reactions as a replacement for structured eval - they are noisy, biased toward users who bother to click, and cannot diagnose root causes. They are a signal, not a verdict.

4. Explanation analysis

4a. General Quality scoring criteria

When the test method is GeneralQuality, Copilot Studio scores the response on 4 distinct criteria. A low General Quality score means one or more of these failed - the customer needs to know WHICH one to fix the right thing:

How the 4 criteria interact: A passing General Quality score means all 4 criteria passed. A failing score means at least one failed - check the explanation field to determine which. The most common failure pattern is Relevance failing alone (knowledge gap), followed by Groundedness failing alone (hallucination). When both Relevance and Groundedness fail together, the agent is likely retrieving the wrong knowledge source entirely.

When NOT to rely on General Quality alone: General Quality checks response quality holistically but cannot verify specific factual values, check tool invocation correctness, or validate structured output formats. Use it alongside targeted methods (CompareMeaning for factual accuracy, ToolUse for action verification, KeywordMatch for required terms).

4b. Explanation pattern mapping

Parse the explanation fields from the CSV. Copilot Studio’s General Quality explanations use these patterns - map each to the criteria above and the Playbook’s diagnostic questions:

For each explanation pattern found in the failures, name the diagnostic area and suggest the specific Playbook question to investigate.

4c. Conversation (multi-turn) result interpretation

When interpreting results from conversation test sets (multi-turn evaluations), the failure patterns differ from single-response tests. Apply these additional diagnostic lenses:

Turn-level diagnosis: A conversation test case fails as a whole, but the root cause is usually in a specific turn. Read the agent's responses turn by turn to locate the first turn where quality degrades. Common patterns:

Available methods are limited: Conversation tests only support General Quality, Keyword Match, Capability Use (Capabilities match), and Custom. If you see failures that would benefit from Compare Meaning or Exact Match analysis (e.g., the agent gave the right answer but phrased differently), note this limitation and recommend the customer also create a complementary single-response test set for those specific scenarios.

Critical turn identification: When reporting failures, identify and call out the critical turn - the specific turn where the conversation went wrong. Downstream turns often fail as a consequence of an earlier turn's failure, not independently. Fixing the critical turn may resolve multiple downstream failures in one change.

4d. Set-level grading interpretation

Copilot Studio’s set-level grading evaluates the test set as a whole - not just aggregating individual pass/fail counts, but assessing overall agent quality across the full set. When the customer has set-level results, interpret them alongside case-level results using this framework:

When set-level and case-level results agree: The straightforward case. A high set-level grade with a high case-level pass rate confirms the agent is performing well. A low set-level grade with many case-level failures confirms systemic problems. Use the standard triage framework above.

When set-level and case-level results diverge - this is where interpretation matters:

How to use set-level grades in the verdict: Set-level grading is supplementary - it does not override the SHIP/ITERATE/BLOCK decision tree, which is based on case-level pass rates by category. However, a low set-level grade on an otherwise SHIP-ready result should trigger a human review checkpoint: “Case-level metrics say SHIP, but set-level quality assessment is below expectations. Review a sample of passing responses before shipping.”

5. Top 3 actions - per the Triage Playbook's Layer 3 (Remediation Mapping)

List exactly three actions in priority order. Each must follow the Playbook's remediation pattern: change X -> re-run Y -> expect Z.

Prioritize using the Playbook's priority order:

1. Safety & compliance failures first
2. Core business failures (highest-frequency query types)
3. Lowest-scoring eval set
4. Recurring failures (same case failing across runs)

Examples of required specificity:

• "Change: Add the product FAQ document to the agent's knowledge sources. Re-run: Cases 4 and 7 (both show 'Question not answered'). Expect: Relevance to pass for product-related queries."
• "Change: Add an escalation instruction to the system prompt: 'If you cannot resolve the request, offer to connect the user with a human agent.' Re-run: Case 3 ('speak to a representative'). Expect: Relevance to pass."
• "Change: Update the expected response in case 5 - it references an outdated process. Re-run: Case 5 only. Expect: Compare Meaning score to improve (this is an eval setup fix, not an agent fix)."

6. Pattern analysis - per the Triage Playbook's Layer 4

Check for these cross-signal patterns from the Playbook:

Also check for concentration: if most failures share a root cause type, call it out. Per the Playbook: "80%+ same root cause = systemic issue, fix the category."

7. Interpretation rationale (teach the WHY)

After presenting the triage, explain the reasoning so the customer can apply this framework independently next time. Cover these four points:

• Why the verdict landed where it did: Walk through the decision tree with the actual numbers. Example: "Safety cases passed at 100% (above the 95% threshold), so we didn't BLOCK. Core business passed at 72% (below the 80% threshold), so the verdict is ITERATE - even though overall pass rate is 78%, the core business shortfall is what drives the decision."
• Why failures were classified the way they were: For each root cause type used, explain the reasoning chain. Example: "Cases 3 and 7 were classified as eval setup issues because the agent's actual response is substantively correct - it answers the question accurately - but the grader rejected it due to phrasing differences. The expected response says 'Contact support at 1-800-555-0100' but the agent says 'You can reach our support team at 1-800-555-0100.' Same information, different words. This is a grader rigidity problem, not an agent problem."
• Why the top 3 actions are in that priority order: Connect each action's priority to the triage framework. Example: "The knowledge source fix is #1 because it addresses 4 of 6 agent failures and they're all core business scenarios. The prompt tweak is #2 because it fixes 2 failures but they're capability scenarios, which rank lower in the Playbook's priority order. The eval fix is #3 because it doesn't improve the agent - it just corrects the measurement."
• What this triage does NOT tell you: Name the limits. Example: "This triage is based on a single eval run. It cannot detect non-determinism issues (run the eval 3 times to check for variance). It also cannot assess whether your test cases cover the right scenarios - a passing eval with poor coverage is worse than a failing eval with good coverage."

This section teaches the methodology so customers can eventually interpret results without the skill. Each bullet must reference the specific data from this eval run, not generic advice.

8. Next-run recommendation

End with one sentence naming exactly what to re-run after making changes. Per the Playbook's re-run targeting:

Tip: After re-running, use Copilot Studio's Result comparison feature to compare the new run against the previous one. It shows which cases flipped pass→fail or fail→pass, making it easy to verify your changes fixed the intended failures without introducing regressions.

8b. Version comparison interpretation (when the customer provides two runs)

If the customer provides results from two eval runs (before/after a change, or two agent configurations), produce a comparison analysis in addition to the standard triage above. Accept this as two CSV files, two pasted summaries, or a description like "Run 1 was 78%, Run 2 is 85%."

Comparison table:

Case-level delta analysis:

Categorize every test case into one of four buckets:

Interpreting deltas:

• +/-5% overall variance between runs is normal (LLM non-determinism). Do not celebrate or panic over small swings. Run the eval 3 times and take the median to distinguish signal from noise.
• A case that flips between runs (pass in one, fail in another, on the same agent version) is a reliability problem, not a quality problem. Flag it separately.
• Regressions outnumbering fixes after a change means the change had a net negative impact, consider reverting.
• All fixes in one category, all regressions in another = instruction conflict. The prompt change that fixed safety responses may have over-constrained business responses. This is the most common pattern when system prompt edits have unintended side effects.

Capability vs. regression framing: Help the customer understand what each eval run type is FOR:

• Capability eval runs target hard scenarios the agent currently fails. Initial pass rates are expected to be low. Success = the pass rate improving over iterations. These are the stretch goals.
• Regression eval runs re-run previously passing test cases after changes. Pass rates should be near 100%. Any drop is a regression that must be investigated. These are the guardrails.

A healthy eval practice uses both: capability evals to push the agent forward, regression evals to ensure it does not slide backward. If the customer is only running one type, recommend adding the other.

Step 3 - Generate output file

After displaying the triage report in conversation, generate a formatted report:

Eval Results Triage Report (.docx) Use the docx skill to create a formatted document containing:

• Title: "Eval Results Triage Report"
• Date and agent name (if known)
• Score summary table
• Verdict (SHIP/ITERATE/BLOCK) with explanation
• Failure triage details for each failing case
• Top 3 prioritized actions
• Pattern analysis
• Interpretation rationale (from section 7 - the WHY behind the verdict, classifications, and priorities)
• Human review checkpoints table (from Step 4)
• Next-run recommendation

Step 4 - Human review checkpoints

After the output file and before the conversation ends, display a Human Review Required section. Eval interpretation is where bad assumptions become bad decisions - a wrong verdict can ship a broken agent or block a good one. These checkpoints flag where human judgment is essential.

Human Review Required

After the checkpoints, add:

• Mandatory reminder: "This triage report was AI-generated from your eval results. Before acting on the verdict or remediation actions, review the failing cases with your team - especially any classified as eval setup issues. The distinction between an agent problem and an eval problem requires human judgment."

Data Retention Warning

Copilot Studio deletes test run results after 89 days. Always recommend that the user:

1. Export the results CSV immediately after each eval run (Test set → Export results)
2. Store alongside the agent version in SharePoint or a repo
3. If the report recommends re-running after fixes, export the current results before changes so before/after comparison is possible

Include this reminder at the end of every generated report.

Behavior rules

• State the verdict FIRST, before any analysis.
• BLOCK immediately if any safety/compliance test case fails. Per the Triage Playbook: safety failures are non-negotiable.
• Always check whether failures are eval setup issues before blaming the agent (Layer 2, Step 1). This is the most common mistake in eval interpretation.
• If pass rate is 100%, treat it as a red flag and say so.
• If input is too sparse for a confident verdict, default to ITERATE and explain why.
• When you cannot determine if a failure is an agent issue or eval setup issue from the CSV alone, say so explicitly and tell the user to read the actualResponse for that row.
• Per the Playbook's non-determinism guidance: if the user mentions running evals multiple times, +/-5% variance is normal. +/-10% requires investigation.

Example invocations

/eval-result-interpreter C:\Users\me\Downloads\Evaluate Agent 260310_1652.csv

/eval-result-interpreter 5/9 passed. Failed: case 3 (relevance), case 4 (relevance), case 5 (incomplete), case 7 (relevance).

/eval-result-interpreter All 8 cases passed on first run.

/eval-result-interpreter [paste CSV contents here]