Agentic Eval

Agentic Evaluation Patterns

Patterns for self-improvement through iterative evaluation and refinement.

Overview

Evaluation patterns enable agents to assess and improve their own outputs, moving beyond single-shot generation to iterative refinement loops.

Generate → Evaluate → Critique → Refine → Output
    ↑                              │
    └──────────────────────────────┘

When to Use

• Quality-critical generation: Code, reports, analysis requiring high accuracy
• Tasks with clear evaluation criteria: Defined success metrics exist
• Content requiring specific standards: Style guides, compliance, formatting

Pattern Selection Guide

Choose the right pattern based on your task:

Patterns

Basic Reflection

Generate → self-critique (PASS/FAIL per criterion as JSON) → refine failed criteria → repeat until all pass or max iterations hit. Non-obvious insight: Always use structured JSON for critique output; free-text critique causes parse failures that break the loop.

Evaluator-Optimizer

Separate generation and evaluation into distinct components: generate(), evaluate() (returns score 0-1 + per-dimension breakdown), optimize() (takes feedback). Stop when overall_score >= threshold. Non-obvious insight: The evaluator must not see the original task prompt to avoid anchoring bias — pass only the output and evaluation criteria.

Code-Specific Reflection

Generate code → generate tests → run tests → fix errors from test output → repeat. Non-obvious insight: Generate tests from the spec, not from the code, to avoid testing implementation details rather than requirements.

Evaluation Strategies

Outcome-Based

Evaluate whether output achieves the expected result.

def evaluate_outcome(task: str, output: str, expected: str) -> str:
    return llm(f"Does output achieve expected outcome? Task: {task}, Expected: {expected}, Output: {output}")

LLM-as-Judge

Use LLM to compare and rank outputs.

def llm_judge(output_a: str, output_b: str, criteria: str) -> str:
    return llm(f"Compare outputs A and B for {criteria}. Which is better and why?")

Rubric-Based

Score outputs against weighted dimensions.

RUBRIC = {
    "accuracy": {"weight": 0.4},
    "clarity": {"weight": 0.3},
    "completeness": {"weight": 0.3}
}

def evaluate_with_rubric(output: str, rubric: dict) -> float:
    scores = json.loads(llm(f"Rate 1-5 for each dimension: {list(rubric.keys())}\nOutput: {output}"))
    return sum(scores[d] * rubric[d]["weight"] for d in rubric) / 5

Rubric + Open Critique (Hybrid)

Rubric-based scoring tracks measurable improvement over time but can only score what the rubric author thought to measure. Pair it with an open critique pass to discover blind spots — important gaps the rubric doesn't cover.

When to use: Any rubric-based eval where the output space is broad enough that a fixed set of dimensions may miss important quality signals (e.g., evaluating a knowledge base, a style guide, or a multi-file skill package).

def open_critique(output: str, rubric_dimensions: list[str]) -> dict:
    """Discover quality gaps a fixed rubric misses."""
    return json.loads(llm(f"""
    You are a senior reviewer. Ignore these rubric dimensions (already scored):
      {rubric_dimensions}
    
    Instead, independently read the output and identify:
    1. blind_spots: Important topics or patterns completely absent
    2. outdated_or_wrong: Advice that is outdated, misleading, or incorrect
    3. structural_issues: Organizational problems (orphaned refs, circular guidance)
    
    Be specific. Do not repeat rubric-territory observations like "needs more examples".
    Return JSON: {{"blind_spots": [...], "outdated_or_wrong": [...], "structural_issues": [...]}}
    
    Output:
    {output}
    """))

def hybrid_evaluate(output: str, rubric: dict) -> dict:
    """Rubric scoring + open critique in one pipeline."""
    # Phase 1: Rubric scores (trackable over time)
    rubric_score = evaluate_with_rubric(output, rubric)
    
    # Phase 2: Open critique (discovery of unknowns)
    critique = open_critique(output, list(rubric.keys()))
    
    # Phase 3: Synthesize — feed BOTH into final report
    report = llm(f"""
    Produce a prioritized improvement plan integrating:
    - Rubric scores: {rubric_score}
    - Open critique findings: {json.dumps(critique)}
    Focus on lowest rubric scores first, then blind spots from the critique.
    """)
    return {"rubric_score": rubric_score, "critique": critique, "report": report}

Key tradeoffs:

When to graduate: Once rubric scores plateau at the ceiling (e.g., all 5/5), the rubric stops being useful. Switch to pairwise LLM-as-Judge comparison against a gold-standard reference, or recalibrate the rubric with harder criteria.

Common Pitfalls

Best Practices

Quick Start Checklist

Evaluation Implementation Checklist

Setup

• Define evaluation criteria/rubric
• Set score threshold for "good enough"
• Configure max iterations (default: 3)

Implementation

• Implement generate() function
• Implement evaluate() function with structured output
• Implement optimize() function
• Wire up the refinement loop
• Consider hybrid approach: rubric + open critique

Safety

• Add convergence detection (stop if no improvement for 2 iterations)
• Log all iterations for debugging
• Handle evaluation parse failures gracefully (regex fallback)
• Harden score parsing (extract first integer, fallback to midpoint)