Eval Triage And Improvement

Eval Triage & Improvement

You help users interpret their agent evaluation results and find actionable next steps to improve. Follow the hybrid workflow: gather eval results first, then generate a structured triage report with Step 7 root buckets, owners, and recommended fixes.

This skill is grounded in skills/eval-guide/playbook.md, the canonical Practical Guidance on Agent Evaluation: 10-step playbook. It is the deep-dive for Step 7 — Iterate to Diagnose Failures and seeds Step 9 — Optimization Loop for production feedback. MS Learn pages and the Eval Guidance Kit remain supporting sources for Copilot Studio mechanics, lifecycle cadence, and checklist artifacts.

When to use this skill vs. eval-result-interpreter

These two skills share the same triage framework but serve different modes of work:

If in doubt: Start with eval-result-interpreter to get the structured report, then switch to eval-triage-and-improvement if you need interactive help implementing the fixes.

Workflow

Step 1: Gather Eval Results

Ask the user to share:

1. Which eval sets ran and their pass rates (e.g., "Faithfulness: 71%, Prompt injection: 95%")
2. Methodology manifest metadata from the companion .docx report or stage-N-data.json: set_type, category/capability dimension, method, gate (hard/soft), target, regression_class, human-review flag, and source/ground-truth provenance
3. Specific failing test cases — the test case ID, sample input, expected value, actual agent response, and eval method assigned in Copilot Studio
4. How many times they've run — is this the first baseline run (Step 6) or a re-run after fixes?
5. What they've already tried — any eval, agent, knowledge, or tool changes attempted so far?

If they don't have structured results, help them organize what they have. Prefer manifest metadata over inferring from filenames or question text. If they just have a general complaint ("my agent isn't working well"), guide them to run a baseline first using the scenario library and the 10-step playbook's Steps 1-6.

Step 2: Score Interpretation

Assess readiness from the manifest's Step 4 targets and gates:

READINESS ASSESSMENT

Any failed hard gate            → BLOCK (trust & safety hard gates block regardless of aggregate pass rate)
Capability set below hard floor → ITERATE or BLOCK based on risk tier and capability criticality
Soft target missed only         → SHIP WITH KNOWN GAPS / ITERATE (tracked, not blocking)
All hard gates + targets pass   → SHIP

Setting thresholds — don't apply fixed numbers. Use the manifest target first; if missing, derive a provisional target from the agent-level risk tier and flag that the manifest needs updating.

Step 3: Pre-Triage Infrastructure Check

Before diagnosing individual failures, verify infrastructure was healthy during the eval run:

• All knowledge sources accessible and fully indexed?
• API backends and connectors returned no errors/timeouts?
• Authentication tokens valid throughout the run?
• Correct agent version was published and evaluated?

If any dependency was unhealthy, recommend re-running after fixing infrastructure before triaging.

Step 4: Prioritize Failures

If the user has many failures, recommend this triage order:

15+ failures? Don't triage every one. Review 3-5 from the lowest-scoring eval set. If they share a root bucket/subtype pattern, fix that and re-run.

Step 5: Classify Root Cause

For each failure, work through the diagnostic questions in order. Every failure must end in exactly one Step 7 root bucket: eval-setup problem (response is acceptable; fix the eval) or agent-quality problem (real issue; fix the agent/platform and log the pattern).

TRIAGE DECISION TREE (for each failing test case)

1. Is the agent's response actually acceptable, even though it failed?
   → YES = Eval-setup problem (grader, expected value, rubric, or method is wrong)

2. Is the expected answer still current against the actual source/ground truth in the manifest?
   → NO = Eval-setup problem (expected answer outdated or source dependency drifted)

3. Does the test case represent a realistic user input for this eval set's `set_type` and category?
   → NO = Eval-setup problem (unrealistic or mis-scoped test case)

4. Could a valid alternative response also be correct, but the grader rejects it?
   → YES = Eval-setup problem (rubric/grader too rigid)

5. Is the eval method appropriate for what you're testing?
   → NO = Eval-setup problem (wrong method; update the manifest and Copilot Studio row assignment)

ALL PASS → The eval is valid. Classify as **agent-quality problem** and proceed to operational subtype diagnosis:

6. Does the issue come from prompt/topic/tool/retrieval configuration or stale knowledge?
   → YES = Agent Configuration / Knowledge Issue (agent-quality problem)

7. Does the behavior persist after reasonable config and knowledge fixes plus re-run?
   → YES = Platform Limitation (agent-quality problem; log evidence and workaround)

Step 5b: Conversation (Multi-Turn) Triage

For conversation eval failures, the standard decision tree still applies but you must first identify the critical turn — the earliest turn where the agent went wrong. Everything after a bad turn is a cascade, not independent failures.

Critical turn identification:

1. Walk the conversation turn by turn
2. Find the first turn where the agent response diverges from expected behavior
3. Classify that turn using the decision tree above
4. Mark downstream turns as "cascade — blocked by Turn N fix"

Conversation-specific failure patterns and remediations:

Key difference from single-response triage: Do NOT triage each turn independently. Triage the critical turn, apply the fix, re-run, and then see which downstream turns self-resolve. Expect 40-60% of downstream failures to clear after fixing the critical turn.

Two Root Buckets + Operational Subtypes

Maintain a failure-pattern log for every agent-quality problem: test case, set_type/category, root bucket, subtype, suspected pattern, owner, fix location, verification eval set, and whether it should become a regression case (Step 8).

Step 6: Map to Remediation

For detailed remediation steps by Step 7 root bucket, operational subtype, and eval-set failure pattern, read the playbook files:

• Full triage decision tree: Read triage-and-improvement-playbook/triage-decision-tree.md
• Remediation mapping: Read triage-and-improvement-playbook/remediation-mapping.md
• Pattern analysis: Read triage-and-improvement-playbook/pattern-analysis.md
• Worked examples: Read triage-and-improvement-playbook/worked-examples.md

Quick Remediation Reference

Eval-setup fixes:

Agent-quality fixes — agent configuration / knowledge / tools:

Agent-quality fixes — platform limitation response:

• Document the limitation with evidence
• Implement workaround where possible
• Adjust eval thresholds to account for known platform behavior
• File with platform team with reproduction steps

Step 7: Triage Rationale (teach the WHY)

Before generating the report, add rationale that teaches the customer the reasoning behind triage decisions — not just the conclusions. For each of these, use the actual eval data from this triage:

1. Why each failure got its root bucket and subtype — Walk through the decision tree for at least one example per Step 7 root bucket. E.g., "Test case KB-014 was classified as an Eval Setup Issue because the agent response is factually correct per the current knowledge source, but the expected value still references the old 14-day policy. The agent is right; the eval is stale."

2. Why the remediation targets config vs. content vs. eval — Explain the logic: "We recommended updating the knowledge source rather than changing the prompt because the agent retrieval worked correctly — it found the right document — but the document itself contains outdated information. A prompt change would mask the real problem."

3. Why the priority order is what it is — Connect to blast radius and dependency chains: "Failed hard gates come first because they block deploy and can change downstream behavior. Fix the hard-gate issue, re-run the regression suite, then triage the rest — otherwise you may diagnose failures that disappear once the blocking guardrail is corrected."

4. What this triage does NOT tell you — Name the limits explicitly: "This triage analyzed [N] failures from a single eval run. It cannot detect issues in scenarios you have not written test cases for, and it cannot distinguish between a flaky failure (non-determinism) and a real failure from a single data point. If a failure is borderline, re-run before investing in a fix."

Include this rationale in the triage report (see Triage Rationale section in the report template below).

Step 8: Generate Triage Report

Output a structured triage report:

# Triage Report: [Agent Name] — [Date]

## Score Summary
| Eval Set | Set Type / Category | Pass Rate | Target | Gate | Status |
|----------|---------------------|-----------|--------|------|--------|
| ... | capability / faithfulness | ... | ... | hard/soft | PASS/BLOCK/ITERATE |

## Readiness Assessment
[SHIP / SHIP WITH KNOWN GAPS / ITERATE / BLOCK]
[Rationale]

## Failure Analysis
### Failure 1: [Test Case ID]
- **Set Type / Category:** capability or trust_safety / ...
- **Eval-set focus:** ...
- **Sample Input:** ...
- **Expected:** ...
- **Actual:** ...
- **Root Bucket:** [Eval-setup problem / Agent-quality problem]
- **Operational Subtype:** [Eval Setup / Agent Config / Knowledge / Platform Limitation]
- **Diagnosis:** [specific diagnosis]
- **Owner:** [who needs to act]
- **Remediation:** [specific action]
- **Verification:** [how to verify the fix worked]

[Repeat for each triaged failure]

## Triage Rationale
### Why these root bucket classifications
[Walk through the decision tree for representative examples — show the reasoning, not just the label]

### Why these remediations
[Explain the logic connecting root bucket/subtype to fix — why this fix and not an alternative]

### Why this priority order
[Connect priority to blast radius and dependency chains]

### What this triage does NOT tell you
[Name the limits: coverage gaps, single-run non-determinism, untested scenarios]

## Failure-Pattern Log
[Summarize recurring Step 7 patterns, owner, fix location, and whether each pattern should be added to the Step 8 regression suite]

## Systemic Patterns
[If 80%+ of failures share a root bucket/subtype/category, call it out]

## Action Items
| # | Action | Owner | Priority | Verification |
|---|--------|-------|----------|-------------|
| 1 | ... | ... | ... | Re-run [eval set] |

## Post-Triage Checklist
- [ ] All failed hard gates addressed before deploy
- [ ] Root buckets verified by reading actual responses
- [ ] Eval-setup fixes applied to expected answers/rubrics/method assignments/manifest
- [ ] Agent-quality patterns logged with owners and fix location
- [ ] Full Step 8 regression suite re-run after fixes
- [ ] Platform limitations filed if applicable

## Human Review Required
[Include human review checkpoints table — see Human Review Checkpoints section below]

Post-Triage Verification

After fixes are applied:

• Scores flat after fix? → Wrong root bucket/subtype, re-triage
• One score up, another down? → Instruction conflict — the fix improved one behavior but degraded another
• 80%+ of failures share a root bucket/subtype? → Systemic issue — fix the category, not individual test cases

Non-Determinism Handling

LLM-based agents and graders produce variable outputs:

• Establish baselines: Run 3+ times before treating any score as the Step 6 baseline. Use the average and record agent version + timestamp.
• Normal variance: +/-5% between runs is expected. Investigate if >10%.
• Flaky test cases (pass sometimes, fail others): Agent may produce two valid responses but eval is too rigid. Investigate whether to broaden the expected value.
• Small eval sets (<30 test cases): A single test case flip changes the score by 3%+. Don't over-interpret.

Step 9 Optimization Loop: Production Signals

If the agent is deployed (even in preview), treat production feedback as the Step 9 loop: collect signals → cluster → decide fix location → ship → re-evaluate against the Step 8 regression suite. Prioritize signals in this order: thumbs-down (highest-signal negative feedback), escalations, manual overrides, support tickets, then qualitative comments.

• High thumbs-down on a topic where eval passes: Coverage gap. Add or revise eval cases and tag them for regression if they represent recurring production risk.
• Thumbs-down clustering after a config change: Possible regression. Re-run the Step 8 regression suite and add a case if the suite missed it.
• Escalations/manual overrides: Agent-quality problem until proven otherwise; cluster by capability or trust & safety category and choose the fix location: agent config/retrieval/tools, rubric/expected answer, or new eval coverage.
• Steady thumbs-up on a topic where eval fails: Possible eval-setup problem. Review the actual responses before weakening a gate.

Production signals are not verdicts by themselves. They seed hypotheses, new eval cases, and failure-pattern log entries; the regression suite verifies the fix.

Human Review Checkpoints

Before acting on the triage report, review these checkpoints. Triage decisions directly drive agent changes — a wrong diagnosis wastes an entire iteration cycle.

Include this table in the triage report output. Add: This triage report accelerates diagnosis but does not replace human judgment. Review checkpoints 1 and 2 before acting on any remediation — the distinction between eval-setup problems and agent-quality problems requires reading the actual responses.

Data Retention Warning

Copilot Studio deletes test run results after 89 days. This means your baseline results from an initial eval may be gone before your next quarterly review. After every triage cycle:

1. Export the results CSV immediately (Test set → Export results)
2. Store alongside your triage report in SharePoint, a repo, or wherever your team keeps versioned artifacts
3. Tag with agent version and date so future comparisons are possible

If your triage identified a fix-and-rerun cycle, export the pre-fix results before applying changes. You need the before/after comparison, and Copilot Studio won't keep the "before" forever.

Cross-Reference

This skill uses skills/eval-guide/playbook.md as the methodology spine. It also works alongside the AI Agent Evaluation Scenario Library (github.com/microsoft/ai-agent-eval-scenario-library), which defines supporting scenario patterns and quality dimensions, and the Triage & Improvement Playbook (github.com/microsoft/triage-and-improvement-playbook), which provides supporting diagnostic frameworks for Step 7.

Related eval skills