AI Eval Regression Tester

AI Eval Regression Tester

When to invoke

• "Run the eval suite for the new prompt version."
• "Compare gpt-X vs the current baseline on our customer-support eval."
• "Block release if eval pass rate drops below 95%."

Inputs needed

1. Eval YAML / JSONL — list of cases with input, expected outputs, graders, tags.
2. Candidate runner — Python callable / HTTP endpoint that takes input and returns output.
3. Baseline run — JSONL of prior outputs (optional, for diffing).
4. Pass thresholds — overall and per-tag.

Workflow

1. Load cases.
2. Run candidate over each case (parallelized).
3. Grade with configured graders.
4. Aggregate — pass rate overall + per tag + diff vs baseline.
5. Gate — exit 1 if any threshold fails (CI-friendly).
6. Report — Markdown + JSONL of every case for diffing.

Eval case schema

- id: refund_basic
  tags: [refund, policy]
  input: "How do I request a refund after 60 days?"
  graders:
    - type: contains
      values: ["return policy", "support team"]
    - type: not_contains
      values: ["sure thing", "lol"]
    - type: regex
      pattern: "policy"
    - type: json_schema
      schema:
        type: object
        required: [answer, citation]

Guardrails

• Always require deterministic graders before LLM-as-judge.
• LLM-judge results must include the judge's reasoning verbatim.
• Per-case results must be logged to JSONL for forensics.
• No silent retries on grader failures.

Reference code

run_eval.py runs cases in parallel, grades, diffs vs baseline, and exits with the right code.