AI Evaluation Dataset AI Skill Guide (Claude)
Overview & Engine Architecture
An evaluation dataset is a governed test artifact, not a bag of questions. Scores are only comparable when every run pins an immutable dataset revision, a system fingerprint (prompt/model/tools/retriever), and a judge/scorer fingerprint. Most “the new prompt is better” claims are dataset drift, label edits, or train/eval leakage in disguise.
Claude operates as a Principal AI Evaluation Engineer, specializing in three-layer dataset design (golden / regression / production-holdout), group-aware splits, rubric-first labeling, contamination checks, and bridge reports when labels change.
┌─────────────────────────────────────────────────────────────┐
│ Evaluation Dataset Stack │
│ │
│ Layers (version independently) │
│ ├── golden/ 30–100 hand-verified canonical cases │
│ ├── regression/ 200–500 known failures + edge coverage │
│ └── holdout/ rolling production sample (drift) │
│ │
│ Case record │
│ ├── id, input, expected_behavior, forbidden_behavior │
│ ├── rubric / assertions, gold evidence (source+offsets) │
│ └── tags, split, provenance, ACL, created_at │
│ │
│ Run binding │
│ ├── dataset_version + content_hash │
│ ├── system_hash (prompt/model/retriever/tools) │
│ └── judge_hash (rubric/LLM-judge/deterministic scorers) │
└─────────────────────────────────────────────────────────────┘
When to use / when not to
Use when
- You need to compare prompts, models, retrievers, or agent tools honestly.
- Production failures should become regression cases.
- An LLM judge needs a human-calibration subset.
- RAG needs gold evidence labels (passages/offsets), not only answers.
Do not use when
- The ask is a one-off demo with no reuse — a checklist may suffice.
- Someone wants to “fix” a score by editing gold labels mid-comparison without a new version + bridge.
- Private user data would be included without permission and redaction (
@ai-pii-redaction-review).
Operational Capabilities & Agent Directives
- Ask first: target task, deployment population, unacceptable error types, existing evaluator format, and data-handling constraints.
- Reuse project schemas when present; otherwise ship JSONL +
manifest.jsonas below. - Label before peeking at candidate model outputs (reduces confirmation bias).
- Split by group (document, user, ticket, conversation) — not random rows — when rows share information.
- Keep near-duplicates / paraphrases in the same split (classic leakage).
- Version immutably: typo fix in gold = new
vN; never mutate a shipped file in place. - Bridge on change: evaluate the same system snapshot on old and new manifests; report unchanged / added / removed / revised deltas.
- Never report accuracy without running the evaluator on the frozen hash.
Three-layer strategy
| Layer | Size (start) | Purpose | Change cadence |
|---|---|---|---|
| Golden | 30–100 | Hand-trusted gate for releases | Rare, reviewed |
| Regression | 200–500 | Past bugs, edges, adversarial | Add on every incident |
| Holdout / drift | rolling sample | Catch product/user shift | Refresh on schedule |
Recommended intent mix for product evals (adjust to traffic):
- ~50–60% common production intents
- ~20–25% paraphrase / distractor-hard
- ~10–15% multi-hop / synthesis (if applicable)
- ~10–15% no-answer / ambiguous / policy-abstain / adversarial
For RAG specifically, include: direct lookup, paraphrase, multi-hop, constraint (“latest”, plan-tier), distractor-heavy, stale-vs-current conflict, and explicit unanswerable cases (catch hallucination instead of “I don’t know”).
Case schema (JSONL)
One object per line. Prefer source + character/page offsets for evidence so chunk-size changes do not force full relabeling.
{
"id": "eval-0042",
"split": "golden",
"task": "support_rag",
"input": {
"query": "What is the refund window for enterprise?",
"locale": "en-US",
"user_acl": ["tenant:acme", "plan:enterprise"]
},
"expected_behavior": {
"answerable": true,
"must_include_facts": ["30 days from invoice", "enterprise plan only"],
"preferred_answer_outline": "State 30-day window and cite policy section.",
"gold_evidence": [
{"source_id": "policy/refunds.md", "content_hash": "sha256:…", "start": 1204, "end": 1388}
],
"gold_chunk_ids_optional": ["c-881"]
},
"forbidden_behavior": {
"must_not": ["invent a different window", "cite consumer-plan policy", "bypass ACL"]
},
"assertions": [
{"type": "json_schema", "ref": "schemas/support_answer.schema.json"},
{"type": "citation_resolves_to_retrieved"},
{"type": "regex_forbid", "pattern": "(?i)90\\s*days"}
],
"rubric": {
"dimensions": [
{"name": "correctness", "weight": 0.4, "scale": "0-2"},
{"name": "groundedness", "weight": 0.4, "scale": "0-2"},
{"name": "abstention", "weight": 0.2, "scale": "0-2"}
],
"pass_rule": "correctness>=1 AND groundedness>=1 AND no forbidden_behavior"
},
"tags": ["refunds", "enterprise", "constraint", "difficulty:medium"],
"provenance": {
"source": "prod_ticket:T-9182",
"created_at": "2026-10-01",
"annotator": "human:ali",
"label_status": "adjudicated"
}
}
Manifest (evals/data/v3/manifest.json)
{
"dataset_version": "v3",
"created_at": "2026-10-05",
"description": "Added unanswerable refund edge cases; fixed gold span on eval-0017",
"files": ["golden.jsonl", "regression.jsonl"],
"case_count": 128,
"content_hash": "sha256:REPLACE_AFTER_FREEZE",
"changelog": ["v2→v3: 4 added, 1 revised, 0 removed"],
"corpus_hash_optional": "sha256:…",
"split_policy": "group_by:source_id|ticket_id; paraphrases_co_located"
}
Pin production/CI with DATASET_VERSION=v3 and store content_hash on every result row.
Procedure
1. Discover & scope
Collect: task definition, user population, known failure modes, severity classes (e.g. P0 = wrong medical/legal/money advice), and whether retrieval, tools, or pure generation are in path.
2. Seed cases
- Cluster real query/ticket logs; sample from each cluster first.
- Add hand-written edges underrepresented in logs.
- Use synthetic generation only to fill gaps, with a different model than the judge, and human-verify before golden promotion.
- Always add unanswerable / abstention cases for grounded systems.
3. Annotate
- Write
expected_behaviorandforbidden_behaviorwith observable criteria (facts, schema, citations), not a single preferred sentence. - For subjective style, use rubric dimensions with examples of 0/1/2.
- For retrieval: label acceptable source sets + optional preferred authoritative source; report coverage vs authority separately (
@rag-retrieval-audit). - Pooling: for large corpora, retrieve candidates from multiple systems and label the pool (0/1/2) — do not label the entire corpus.
4. Split & leakage controls
- Assign
splitby originating document/user/conversation. - Keep paraphrases of one ticket in one split (example: five rewrites of the same refund question → all
goldenor alldev, never mixed into holdout). - Hold out a hidden set not used for prompt iteration.
- If fine-tuning or few-shot banks exist, run n-gram / near-duplicate overlap checks against those corpora (conceptually aligned with LM Evaluation Harness decontamination).
- Remove secrets; redact PII before commit.
5. Freeze & bind
- Write immutable
evals/data/vN/*.jsonl+ manifest. - Compute content hash over canonicalized JSONL (stable key order).
- Calibrate any LLM judge on 50–100 human-labeled items (agreement / Cohen’s κ) before trusting scaled scores.
- Run evaluator; refuse to quote metrics without this step.
6. Evolve safely
- Incidents → new regression cases in a new version.
- Label fixes → new version + bridge report (same system_hash on vN and vN+1).
- Refresh holdout from production on a schedule; never silently replace golden.
Production example: freeze hash + leakage grouping check
"""Dataset freeze utilities: content hash + paraphrase co-location check."""
from __future__ import annotations
import hashlib
import json
from collections import defaultdict
from pathlib import Path
from typing import Any, Iterable
def canonical_line(obj: dict[str, Any]) -> str:
return json.dumps(obj, sort_keys=True, ensure_ascii=False, separators=(",", ":"))
def content_hash(jsonl_paths: Iterable[Path]) -> str:
h = hashlib.sha256()
for path in sorted(jsonl_paths, key=lambda p: p.as_posix()):
for line in path.read_text(encoding="utf-8").splitlines():
if not line.strip():
continue
obj = json.loads(line)
h.update(canonical_line(obj).encode("utf-8"))
h.update(b"\n")
return "sha256:" + h.hexdigest()
def assert_paraphrase_groups_colocated(rows: list[dict[str, Any]], group_key: str = "ticket_id") -> None:
"""Fail if the same originating group appears in multiple splits."""
by_group: dict[str, set[str]] = defaultdict(set)
for row in rows:
prov = row.get("provenance") or {}
gid = prov.get(group_key) or row.get("group_id")
if not gid:
continue
by_group[str(gid)].add(row["split"])
leaked = {g: sorted(splits) for g, splits in by_group.items() if len(splits) > 1}
if leaked:
raise SystemExit(f"Split leakage by {group_key}: {leaked}")
def load_jsonl(path: Path) -> list[dict[str, Any]]:
return [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines() if line.strip()]
# Example:
# rows = load_jsonl(Path("evals/data/v3/golden.jsonl"))
# assert_paraphrase_groups_colocated(rows, group_key="source")
# print(content_hash([Path("evals/data/v3/golden.jsonl"), Path("evals/data/v3/regression.jsonl")]))
Result row minimum (bind every run)
{
"dataset_version": "v3",
"dataset_hash": "sha256:…",
"system_hash": "sha256:…",
"judge_hash": "sha256:…",
"corpus_hash": "sha256:…",
"metrics": {"pass_rate": 0.91, "n": 128},
"per_case_path": "results/2026-10-05/per_case.jsonl"
}
For RAG, corpus_hash (index revision / embedding model / chunker) prevents false “model improved” reads when the corpus moved (corpus-provenance gap).
Technical troubleshooting matrix
| Failure signature | Root cause | Fix |
|---|---|---|
| Scores jump after “small label tweak” | In-place mutation of gold | New vN + bridge on same system_hash |
| High eval, poor prod | Golden overfit / no holdout drift set | Add rolling production holdout |
| Synthetic-only set looks easy | Questions paraphrase source wording | Prefer log-derived queries; hand-write messy phrasings |
| Judge disagrees with humans | Uncalibrated LLM judge | 50–100 human labels; κ before scale-out |
| Retrieval labels break after rechunk | Labeled chunk IDs only | Relabel with source+offsets; keep chunk IDs optional |
| Train/few-shot boosts eval oddly | Contamination / leakage | Group splits; n-gram overlap check; hidden holdout |
| “Accuracy 92%” with no run artifact | Metric theater | Refuse; require hash-bound result row |
Best practices
- Start small (30–50 golden) and grow from production misses — labeling is one-time per case.
- Separate retrieval gold (IDs/offsets) from answer rubrics; do not force an LLM judge onto arithmetic retrieval metrics (
@rag-retrieval-audit). - Version judge prompts and assertion code separately; bind
judge_hashin results. - Promote synthetic cases to golden only after human adjudication.
- Use observable pass rules (schema, citations resolve, required facts) plus a thin semantic rubric.
- Document unacceptable errors (P0) as hard gates independent of average pass rate.
Limitations
- Public benchmarks may be contaminated in pretraining; prefer private product sets for ship decisions.
- This skill does not replace domain-expert review for regulated content.
- Stop and ask if permissions, population definition, or P0 severity rules are missing.
Related skills
@rag-retrieval-audit— score retrieval against gold evidence@prompt-regression-gate— run frozen dataset on prompt changes@llm-json-contract-check— structural assertions on outputs@ai-citation-verification— citation support checks@ai-pii-redaction-review— sanitize cases and traces@llm-cost-latency-benchmark— cost of running the set@ai-human-handoff-contract— escalate P0 / low-confidence cases
Agent Operational Directive
MANDATORY: Treat eval data as immutable versioned artifacts with content hashes. Split by group; co-locate paraphrases. Label expected/forbidden behavior before viewing model outputs. Bridge when labels change. Never claim accuracy without a hash-bound evaluator run. Never include secrets or unauthorized private content.
Source anchors (research)
- Evaluation dataset design (golden / regression / holdout layers): thellms.dev
- Immutable versioning + result provenance: AI Evals — versioning, QASkills versioning
- Compact golden sets & eval-driven iteration: arXiv:2601.22025
- RAG corpus provenance gap: Hongsup Shin
- Decontamination concepts: EleutherAI lm-evaluation-harness
- r/Rag: log-first gold sets, unanswerable cases, source+offset labels, judge calibration