majiayu000 AgentV Eval Analyzer
Analyze JSONL evaluation results and produce actionable suggestions for improving eval quality, reducing cost, and increasing reliability.
When to Use
- After running
agentv evaland wanting to improve your evaluation config - When evaluations are slow or expensive and you suspect LLM-judges are doing deterministic work
- When reviewing eval quality before sharing or publishing evaluation files
- When triaging flaky evaluations that produce inconsistent scores
Quick Start
# Find your most recent results file
agentv trace list --limit 5
# Dispatch the eval-analyzer agent with the results file
# Agent: eval-analyzer
# Parameters:
# results-file: <path-to-results.jsonl>
# eval-path: <path-to-eval.yaml> (optional, for deeper analysis)
Dispatch the eval-analyzer agent with:
results-file: Path to the JSONL results (from.agentv/results/oragentv trace list)eval-path(optional): Path to the EVAL.yaml for assertion-level analysis
The agent produces a read-only report — it never modifies files.
What It Detects
1. Deterministic-Upgrade Candidates (highest value)
LLM-judge evaluators doing work that a deterministic assertion could handle — cheaper, faster, and more reliable.
| Pattern in LLM-Judge Reasoning | Suggested Deterministic Type |
|---|---|
| "Output contains 'X'" — always cites same substring | type: contains, value: "X" |
| Score always 0 or 1, never partial — binary check | type: equals or specific deterministic |
| "Response is valid JSON" — format validation | type: is-json |
| "Output starts with 'Error:'" — prefix check | type: regex, value: "^Error:" |
| "Matches pattern /regex/" — regex match | type: regex, value: "/pattern/" |
| All passed assertions are substring presence checks | Multiple type: contains assertions (one per value) |
2. Weak Assertion Detection
| Weakness | Example | Fix |
|---|---|---|
| Vague (< 8 words, no specifics) | "Response is good" | Add measurable criteria |
| Tautological | "Output is correct" | Define what "correct" means with expected values |
| Compound | "Handles errors and returns JSON" | Split into separate assertions |
| Overly broad LLM-judge | Single vague prompt string |
Use type: rubrics with enumerated items |
3. Cost/Quality Flags
| Flag | Meaning |
|---|---|
| Expensive binary check | LLM-judge always returns 0 or 1 → deterministic replacement |
| Always-pass | Score 1.0 on every test → criteria may be too lenient |
| Always-fail | Score 0.0 on every test → criteria may be misconfigured |
| Redundant evaluators | Two evaluators with identical scores/reasoning → merge |
4. Multi-Provider Variance
When results span multiple targets, flags evaluators with > 0.3 score variance across providers — indicating provider-sensitive assertions that may need tightening.
Applying Suggestions
The analyzer report includes concrete YAML snippets for each suggestion. To apply:
- Open the EVAL.yaml referenced in the report
- Find the
assertionsentry for the flagged evaluator (matched bynameandtest_id) - Replace or supplement the evaluator config with the suggested deterministic assertion
- Re-run
agentv evalto verify the change produces equivalent scores
Example upgrade:
Before (LLM-judge doing substring work):
assertions:
- name: has-error-code
type: llm-judge
prompt: "Check if the response contains the error code 404"
After (deterministic, zero LLM cost):
assertions:
- name: has-error-code
type: contains
value: "404"
Limitations
- Deterministic-upgrade detection is heuristic — review suggestions before applying
- Requires at least one completed eval run (needs JSONL results)
- Cannot detect all weak assertions without the EVAL.yaml file (results-only mode has reduced coverage)
- Multi-run comparison (flakiness across runs) is out of scope — use
agentv comparefor that