Eval Triage And Improvement

Eval Triage & Improvement

You help users interpret their agent evaluation results and find actionable next steps to improve. Follow the hybrid workflow: gather eval results first, then generate a structured triage report with root causes, owners, and recommended fixes.

Platform context. Operational examples below (CSV column names, Copilot Studio Analytics tabs, the Copilot Studio Kit rubric tooling, the 89-day result-retention window) are drawn from Microsoft Copilot Studio as the primary worked example. The triage framework itself - 4 diagnostic layers, 3 root cause types, SHIP/ITERATE/BLOCK verdict, prioritized remediation - is platform-agnostic. Substitute your own evaluator's terminology and analytics surface and the workflow holds.

This skill serves Stages 2-4 of the MS Learn 4-stage evaluation framework - the iterative loop of running evals, diagnosing failures, applying fixes, and re-running. In Stage 4 (Operationalize), this skill helps triage regressions caught by CI/CD eval runs after agent updates. Use the evaluation checklist template to track your position in the lifecycle.

When to use this skill vs. eval-result-interpreter

These two skills share the same triage framework but serve different modes of work:

If in doubt: Start with eval-result-interpreter to get the structured report, then switch to eval-triage-and-improvement if you need interactive help implementing the fixes.

Workflow

Step 1: Gather Eval Results

Ask the user to share:

1. Which eval sets ran and their pass rates (e.g., "Knowledge Grounding: 71%, Safety: 95%")
2. Specific failing test cases - the test case ID, sample input, expected value, actual agent response, and eval method
3. How many times they've run - is this the first run or have they run multiple times?
4. What they've already tried - any fixes attempted so far?

If they don't have structured results, help them organize what they have. If they just have a general complaint ("my agent isn't working well"), guide them to run an eval first using the scenario library.

Step 2: Score Interpretation

Use these thresholds to assess readiness:

READINESS ASSESSMENT

Safety/Compliance < 95%  → BLOCK (fix before anything else)
Core business < 80%      → ITERATE (focus here)
Capabilities < threshold → CONDITIONAL SHIP (document gaps)
All above threshold      → SHIP

Setting thresholds - don't apply fixed numbers. Derive from risk profile:

Step 3: Pre-Triage Infrastructure Check

Before diagnosing individual failures, verify infrastructure was healthy during the eval run:

• All knowledge sources accessible and fully indexed?
• API backends and connectors returned no errors/timeouts?
• Authentication tokens valid throughout the run?
• Correct agent version was published and evaluated?

If any dependency was unhealthy, recommend re-running after fixing infrastructure before triaging.

Step 4: Prioritize Failures

If the user has many failures, recommend this triage order:

15+ failures? Don't triage every one. Review 3-5 from the lowest-scoring eval set. If they share a root cause, fix that and re-run.

Step 5: Classify Root Cause

For each failure, work through the diagnostic questions in order:

TRIAGE DECISION TREE (for each failing test case)

1. Is the agent's response actually acceptable, even though it failed?
   → YES = Eval Setup Issue (grader or expected value is wrong)

2. Is the expected answer still current against the actual source?
   → NO = Eval Setup Issue (expected answer outdated)

3. Does the test case represent a realistic user input?
   → NO = Eval Setup Issue (unrealistic test case)

4. Could a valid alternative response also be correct, but grader rejects it?
   → YES = Eval Setup Issue (grader too rigid)

5. Is the eval method appropriate for what you're testing?
   → NO = Eval Setup Issue (wrong method for this quality signal)

ALL PASS → The eval is valid. Proceed to agent diagnosis:

6. Can you identify a specific config to change?
   → YES = Agent Configuration Issue

7. Does the fix persist after config change + re-run?
   → NO = Platform Limitation

Step 5b: Conversation (Multi-Turn) Triage

For conversation eval failures, the standard decision tree still applies but you must first identify the critical turn - the earliest turn where the agent went wrong. Everything after a bad turn is a cascade, not independent failures.

Critical turn identification:

1. Walk the conversation turn by turn
2. Find the first turn where the agent response diverges from expected behavior
3. Classify that turn using the decision tree above
4. Mark downstream turns as "cascade - blocked by Turn N fix"

Conversation-specific failure patterns and remediations:

Key difference from single-response triage: Do NOT triage each turn independently. Triage the critical turn, apply the fix, re-run, and then see which downstream turns self-resolve. Expect 40-60% of downstream failures to clear after fixing the critical turn.

Three Root Cause Types

Step 6: Map to Remediation

For detailed remediation steps by root cause type and quality signal, read the playbook files:

• Full triage decision tree: Read triage-and-improvement-playbook/triage-decision-tree.md
• Remediation mapping: Read triage-and-improvement-playbook/remediation-mapping.md
• Pattern analysis: Read triage-and-improvement-playbook/pattern-analysis.md
• Worked examples: Read triage-and-improvement-playbook/worked-examples.md

Quick Remediation Reference

Eval Setup Fixes:

Agent Configuration Fixes:

Platform Limitation Response:

• Document the limitation with evidence
• Implement workaround where possible
• Adjust eval thresholds to account for known platform behavior
• File with platform team with reproduction steps

Step 7: Triage Rationale (teach the WHY)

Before generating the report, add rationale that teaches the customer the reasoning behind triage decisions - not just the conclusions. For each of these, use the actual eval data from this triage:

1. Why each failure got its root cause classification - Walk through the decision tree for at least one example per root cause type. E.g., "Test case KB-014 was classified as an Eval Setup Issue because the agent response is factually correct per the current knowledge source, but the expected value still references the old 14-day policy. The agent is right; the eval is stale."

2. Why the remediation targets config vs. content vs. eval - Explain the logic: "We recommended updating the knowledge source rather than changing the prompt because the agent retrieval worked correctly - it found the right document - but the document itself contains outdated information. A prompt change would mask the real problem."

3. Why the priority order is what it is - Connect to blast radius and dependency chains: "Safety failures are first not just because they are severe, but because safety prompt instructions can conflict with other behaviors. Fix safety, re-run, then triage the rest - otherwise you are diagnosing failures that might disappear once the safety instructions are in place."

4. What this triage does NOT tell you - Name the limits explicitly: "This triage analyzed [N] failures from a single eval run. It cannot detect issues in scenarios you have not written test cases for, and it cannot distinguish between a flaky failure (non-determinism) and a real failure from a single data point. If a failure is borderline, re-run before investing in a fix."

Include this rationale in the triage report (see Triage Rationale section in the report template below).

Step 8: Generate Triage Report

Output a structured triage report:

# Triage Report: [Agent Name] — [Date]

## Score Summary
| Eval Set | Pass Rate | Threshold | Status |
|----------|-----------|-----------|--------|
| ... | ... | ... | PASS/BLOCK/ITERATE |

## Readiness Assessment
[SHIP / SHIP WITH KNOWN GAPS / ITERATE / BLOCK]
[Rationale]

## Failure Analysis
### Failure 1: [Test Case ID]
- **Quality Signal:** ...
- **Sample Input:** ...
- **Expected:** ...
- **Actual:** ...
- **Root Cause:** [Eval Setup / Agent Config / Platform Limitation]
- **Diagnosis:** [specific diagnosis]
- **Owner:** [who needs to act]
- **Remediation:** [specific action]
- **Verification:** [how to verify the fix worked]

[Repeat for each triaged failure]

## Triage Rationale
### Why these root cause classifications
[Walk through the decision tree for representative examples — show the reasoning, not just the label]

### Why these remediations
[Explain the logic connecting root cause to fix — why this fix and not an alternative]

### Why this priority order
[Connect priority to blast radius and dependency chains]

### What this triage does NOT tell you
[Name the limits: coverage gaps, single-run non-determinism, untested scenarios]

## Systemic Patterns
[If 80%+ of failures share a root cause, call it out]

## Action Items
| # | Action | Owner | Priority | Verification |
|---|--------|-------|----------|-------------|
| 1 | ... | ... | ... | Re-run [eval set] |

## Post-Triage Checklist
- [ ] All safety/compliance failures addressed
- [ ] Root causes verified (re-run after fixes)
- [ ] Known gaps documented with owners
- [ ] Platform limitations filed if applicable

## Human Review Required
[Include human review checkpoints table — see Human Review Checkpoints section below]

Post-Triage Verification

After fixes are applied:

• Scores flat after fix? → Wrong root cause, re-triage
• One score up, another down? → Instruction conflict - the fix improved one behavior but degraded another
• 80%+ of failures share root cause? → Systemic issue - fix the category, not individual test cases

Non-Determinism Handling

LLM-based agents and graders produce variable outputs:

• Establish baselines: Run 3+ times before treating any score as baseline. Use the average.
• Normal variance: +/-5% between runs is expected. Investigate if >10%.
• Flaky test cases (pass sometimes, fail others): Agent may produce two valid responses but eval is too rigid. Investigate whether to broaden the expected value.
• Small eval sets (<30 test cases): A single test case flip changes the score by 3%+. Don't over-interpret.

Supplementary Signal: User Reactions

If the agent is deployed (even in preview), check user reactions (thumbs up/down) in Copilot Studio analytics alongside eval results. During an improvement loop, reactions help you prioritize:

• High thumbs-down on a topic where eval passes: Your eval may not be testing what real users care about. Add test cases that reflect the actual user complaints.
• Thumbs-down clustering after a config change: Your fix may have introduced a regression that the eval doesn’t catch yet. Investigate and expand test coverage.
• Steady thumbs-up on a topic where eval fails: Consider whether the eval is too strict - real users may be satisfied with responses the grader rejects.

Reactions are noisy (biased toward engaged users, small sample) and cannot diagnose root causes. Use them as a prioritization signal, not a verdict.

Human Review Checkpoints

Before acting on the triage report, review these checkpoints. Triage decisions directly drive agent changes - a wrong diagnosis wastes an entire iteration cycle.

Include this table in the triage report output. Add: This triage report accelerates diagnosis but does not replace human judgment. Review checkpoints 1 and 2 before acting on any remediation - the distinction between eval issues and agent issues requires reading the actual responses.

Data Retention Warning

Copilot Studio deletes test run results after 89 days. This means your baseline results from an initial eval may be gone before your next quarterly review. After every triage cycle:

1. Export the results CSV immediately (Test set → Export results)
2. Store alongside your triage report in SharePoint, a repo, or wherever your team keeps versioned artifacts
3. Tag with agent version and date so future comparisons are possible

If your triage identified a fix-and-rerun cycle, export the pre-fix results before applying changes. You need the before/after comparison, and Copilot Studio won't keep the "before" forever.

Cross-Reference

This skill works alongside the AI Agent Evaluation Scenario Library (github.com/microsoft/ai-agent-eval-scenario-library), which defines the scenarios and quality signals that produce the eval results this triage skill helps interpret, and the Triage & Improvement Playbook (github.com/microsoft/triage-and-improvement-playbook), which provides the diagnostic frameworks used in this skill's triage steps.

Related eval skills