Eval

Eval

Measure before you ship. An LLM that feels better may not score better. Verify with evidence.

Evaluation Modes

Automated Metrics

Use for text generation quality:

• ROUGE-L: recall-oriented overlap — good for summarization
• BLEU: precision-oriented n-gram overlap — good for translation
• BERTScore: semantic similarity via embeddings — better than n-gram overlap
• Exact Match / F1: for extractive QA (spans, named entities)
• Pass@K: for code generation — does any of K samples pass tests?

Score threshold: ROUGE-L > 0.4, BERTScore F1 > 0.85 for production tasks

LLM-as-Judge

Ask a capable model (e.g., Claude Opus) to score outputs:

Rate the following response on a scale of 1–5 for:
- Correctness: is the answer factually accurate?
- Completeness: does it address all parts of the question?
- Conciseness: is it appropriately brief?

Question: <original prompt>
Response: <model output>
Reference: <expected answer>

Return JSON: {"correctness": N, "completeness": N, "conciseness": N, "reasoning": "..."}

Rules:

• Use a different model as judge than the one being evaluated (avoids self-serving bias)
• Always include a reference answer when available
• Run each judgment 3× and average (judges are noisy)

Regression Suite

Build and maintain a golden dataset:

1. Capture: after each manual review, save {input, expected_output, tags} to evals/golden.jsonl
2. Run: on every PR, score all golden cases and report delta
3. Gate: fail PR if regression > 5% on any dimension
4. Evolve: add new cases for every fixed bug or edge case found

eval/
├── golden.jsonl       — curated input/output pairs
├── run.ts             — eval runner (loads golden, calls model, scores)
└── report.md          — latest results (committed, not gitignored)

Benchmark Suites (Public)

For capability measurement against known standards:

• MMLU — broad knowledge (57 subjects)
• GSM8K — multi-step math reasoning
• HumanEval / MBPP — code generation
• TruthfulQA — hallucination resistance
• HellaSwag — commonsense reasoning

Use lm-evaluation-harness (EleutherAI) to run standardized benchmarks.

Steps

1. Define what "good" means — pick 2–3 dimensions that matter for your use case
2. Collect 50+ representative inputs (cover happy path, edge cases, adversarial)
3. Run baseline (current model/prompt) and record scores
4. Make your change (new prompt, model, RAG config)
5. Re-run and compare — report delta, not just absolute scores
6. Commit passing evals to golden suite; add failing cases as regression tests

Output

Eval Report
───────────
Suite:       <golden | benchmark name>
Cases:       <N total>
Dimensions:  <correctness, completeness, ...>
Baseline:    <scores>
Current:     <scores>
Delta:       <+/- per dimension>
Regressions: <list of failing cases if any>
Decision:    PASS | FAIL