# AI Evals > Designs trustworthy LLM/agent evals and optimization loops. Use when building graders, calibrating judges, choosing eval/fine-tune methods, thresholds, or fixing noisy scores. - Skill: `majiayu000/ai-evals` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds add majiayu000/ai-evals` - Raw SKILL.md: https://api.skillmd.com/api/skills/majiayu000/ai-evals/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: majiayu000 (https://skillmd.com/u/majiayu000) - Updated: 2026-09-09 - Page: https://skillmd.com/skills/majiayu000/ai-evals --- # AI Evaluation and Fine-Tuning Methodology Skill **Core stance**: an eval is an instrument. An untrusted instrument is worse than no instrument, because it produces confident wrong numbers that ship regressions. Fine-tuning is an optimization loop around that instrument. If the instrument is weak, training just makes the model better at gaming bad measurement. This skill is the cross-domain methodology layer that domain eval and model-lifecycle skills defer to: how to keep an LLM-as-judge honest, integrate eval frameworks, choose between prompting/context/tools/test-time compute/SFT/preference/RFT/PEFT/ distillation, derive thresholds instead of guessing them, and stop flaky runs or training leakage from masquerading as progress. This is the **methodology umbrella** for evals. Domain skills own *what* to measure; this skill owns *whether you can trust the measurement*. - Building an eval system for a **coding agent** -> [ai-coding-agents-observability-evals](../ai-coding-agents-observability-evals/SKILL.md) - Evaluating **RAG / retrieval / search** -> [ai-rag](../ai-rag/SKILL.md) - Running **Hub model benchmarks** (inspect-ai, lighteval) -> use the `huggingface-skills:` plugin (external) - General **LLM lifecycle** decisions -> [ai-llm](../ai-llm/SKILL.md) - This skill: **judge bias, framework choice, calibration, reproducibility, optimization technique gates** — the parts those four share and none owns in depth. ## ASCII Flow ```text eval need | v define verifiable goal (what would FAIL if the requirement reverted?) | v choose grader deterministic check -> LLM-as-judge -> human label (cheapest that works) | v control judge bias position / length / self-preference / verbosity | v derive thresholds from a labeled calibration set (not vibes) | v choose optimization path (prompt/RAG/tools -> SFT -> preference/RFT/PEFT) | v choose inference-time lift (self-consistency / rerank / verify / refine) | v control flake (pass@k, low temp, quarantine unstable cases) | v trustworthy gate -> train / block / ship / rollback ``` ## Quick Reference | Task | Read or Run | Outcome | |------|-------------|---------| | Build a (question, ideal-answer) set and tune it | `references/dataset-construction.md` | Sourcing, ideal-answer authoring, run→compare→tune loop | | Stop a judge from rating its own output high | `references/llm-judge-bias.md` | Self-preference, position, length, verbosity controls | | Pick / wire an eval framework | `references/framework-integration.md` | inspect-ai, lighteval, Ragas, DeepEval, promptfoo, Braintrust integration snippets + when to use each | | Choose a pass threshold defensibly | `references/threshold-derivation.md` | Derive thresholds from a labeled set; inter-rater agreement; gate design | | Stop flaky runs reading as regressions | `references/flake-and-reproducibility.md` | pass@k, seeds, temperature, quarantine, contamination/leakage | | Decide if "A beats B" is real, size the set | `references/eval-statistics.md` | Bootstrap CIs, McNemar, power/MDE sizing, FDR, variance reduction | | Get maximum from an LLM | `references/llm-optimization-technique-map.md` | Technique ladder across prompts, data, RAG/tools, test-time compute, SFT, preference/RFT, PEFT, distillation | | Decide whether and how to fine-tune | `references/fine-tuning-eval-loop.md` | Prompt/RAG/tool baseline, SFT vs preference/RFT vs PEFT, split hygiene, promotion gates | | Evaluate on live/production traffic | `references/online-production-eval.md` | Offline-online correlation, A/B+guardrails, shadow/canary, drift, regression replay, HITL | | Evaluate refusals, jailbreaks, harm | `references/safety-redteam-eval.md` | Over/under-refusal, ASR per attack family, injection, harm rubrics, robustness | | Go beyond one judge | `references/advanced-judging.md` | Juries, fine-tuned judges, CoT/probability scoring, calibration (kappa/ECE), agentic reward | ## When to Use This Skill Activate when the user asks for: - An LLM-as-judge / LLM grader and how to keep it honest - Why eval scores look inflated, noisy, or contradictory - Which eval framework to use, or how to integrate one - How to set or justify a pass/fail threshold or release gate - Pairwise / preference evaluation between two prompts, models, or harnesses - Reducing eval flakiness, contamination, or testset leakage - Calibrating a judge against human labels - Deciding whether to fine-tune, how to select SFT vs preference/RFT vs PEFT, or whether a fine-tuned model is genuinely better than a prompt/RAG/tool baseline - Getting maximum performance from an LLM using known techniques, including prompt/context/tool changes, test-time compute, reranking, distillation, or post-training ## Scope Boundaries (Use These Skills for Depth) - **Domain metrics for retrieval** (nDCG/MRR/recall, faithfulness) -> [ai-rag](../ai-rag/SKILL.md) - **Agent golden tasks, tool-call grading, cost ops** -> [ai-coding-agents-observability-evals](../ai-coding-agents-observability-evals/SKILL.md) - **Running benchmark harnesses on Hub models** -> use the `huggingface-skills:` plugin (external) - **Prompt CI/CD and structured output contracts** -> [ai-prompt-engineering](../ai-prompt-engineering/SKILL.md) - **General model selection, serving, quantization, and deployment economics** -> [ai-llm](../ai-llm/SKILL.md) and [ai-llm-inference](../ai-llm-inference/SKILL.md) ## Workflow 1. **Transform the vague ask into a verifiable goal.** "Is it good?" is not gradeable. Ask: which case would fail first if the requirement reverted? 2. **Build the dataset before the grader.** Source real questions, author ideal answers from the system's allowed context, and plan the run→compare→tune loop — see `references/dataset-construction.md`. No dataset, no eval. 3. **Pick the cheapest grader that works.** Deterministic check > LLM judge > human. Reserve the LLM judge for what code cannot decide (Rule 5: use the model only for judgment calls). 4. **If using an LLM judge, control its bias** before trusting any number — see `references/llm-judge-bias.md`. Untreated judge bias is the #1 source of confidently-wrong eval scores. When one judge isn't enough (high stakes, weak agreement, open-ended), escalate to juries / fine-tuned judges / calibrated scoring — see `references/advanced-judging.md`. 5. **Control flake and leakage** with pass@k, low judge temperature, seed pinning, and held-out testsets — see `references/flake-and-reproducibility.md`. 6. **Derive thresholds from a labeled calibration set**, not intuition or copied targets — see `references/threshold-derivation.md`. **Size the gating set and judge "A beats B" with statistics** (bootstrap CIs, McNemar, power/MDE, FDR) — see `references/eval-statistics.md`. A score difference without a CI is not a result. 7. **Only fine-tune after the baseline has earned it.** Compare prompt/RAG/tool fixes first, then choose SFT for imitation/style/format/tool-call behavior, preference/RFT for rubric-scored reasoning or tradeoffs, and PEFT/LoRA/QLoRA when adapting an open model under compute or deployment constraints — see `references/fine-tuning-eval-loop.md`. Training loss is telemetry; held-out behavior is the verdict. 8. **Apply the full optimization ladder, not one pet method.** For maximum LLM performance, evaluate cheap prompt/context/tool fixes, then inference-time methods (self-consistency, best-of-N, rerank/verify/refine), then data/SFT, preference/RFT/RLVR, PEFT, and distillation as the evidence warrants — see `references/llm-optimization-technique-map.md`. Each technique gets its own failure mode and gate. 9. **Gate loudly.** A gate that passes while silently skipping cases is a failure dressed as success (Rule 12: fail loud). Report skipped/quarantined cases in the gate output. 10. **Extend past the offline gate where the system warrants it.** Add safety/red-team evaluation (refusal precision/recall, jailbreak ASR, injection, harm rubrics) — see `references/safety-redteam-eval.md` — and, once in production, online evaluation (offline-online correlation, A/B with guardrails, drift, regression replay) — see `references/online-production-eval.md`. The offline gate is a filter; production is the verdict. ## Core Principles - **The judge is a model with failure modes.** Treat its scores as one calibrated input, never as ground truth. - **Different judge than the one under test.** Self-preference bias is real and large; never gate on a model grading its own family/config. - **Behavior, not plausibility.** Rubrics that reward "looks good" reward length and confidence. Pin rubrics to verifiable behavior. - **No threshold without a labeled set.** A copied target (">95%") is a guess until validated on your own distribution. - **No fine-tune without a baseline and a holdout.** A tuned model that beats no prompt/RAG/tool baseline, or only wins on the training/dev set, has not earned release. - **No "maximum performance" without a technique ladder.** The best result often comes from composition: cleaner data + stronger retrieval/tool contracts + calibrated judge + small test-time search + selective post-training. Test the cheapest credible lift before moving weights. - **Optimize behavior, not hidden knowledge.** Fine-tune for stable formatting, domain style, tool-use patterns, rubric-following, or compact specialized behavior. Use retrieval/context for facts that change or must be cited. - **Flake is a broken test, not a regression.** A verdict that flips run-to-run means the eval is wrong, not the system. - **Held-out or it's contaminated.** If tuning ever saw the eval cases, the scores are inflated. - **Goodhart's Law is the default outcome, not an edge case.** Any metric that becomes a target (a threshold, a bonus, a promotion gate) will eventually be gamed — by the system under test, by whoever tunes against it, or by the judge itself. Every trap and anti-pattern in this file is a specific instance of this one law; treat a metric that stops correlating with the outcome you actually care about as expected decay, and re-anchor it against production outcomes or fresh human judgment on a schedule, not only when someone notices. - **A point estimate is not a result.** Every reported number in this skill's gates — pass rate, win rate, judge-human agreement — is a sample statistic with sampling error. Report it with a confidence interval or it is not reportable; see `references/eval-statistics.md`. ## Known Traps - Grading an agent with the same model that produced the output (self-preference) - Comparing two candidates in fixed order and trusting the winner (position bias) - Copying a `>95%` threshold from a blog without validating it on your data - One judge call per request with no cheap deterministic pre-filter (cost blowup) - Treating a run-to-run verdict flip as a real regression instead of quarantining - Generating a synthetic testset from the same docs used to tune the system - Reporting "all passed" when some cases were skipped or errored (silent success) - Claiming "A beats B" from a point estimate with no confidence interval or test - Gating a small regression on a set far too small to detect it (no power check) - Tuning safety to block harm without a benign set, so the model over-refuses - Trusting a seed for reproducibility through a hosted API that isn't deterministic - Fine-tuning because the prompt is messy, the retrieval is broken, or the tool contract is ambiguous - Declaring the fine-tune better from training loss, validation loss, or one cherry-picked demo instead of a paired held-out eval with CIs - Letting the training set, grader calibration set, and release gate share cases - Training a judge or reward model on labels produced only by the same model family it will later grade ## Common Anti-Patterns - Vibes-based eval: spot-checking a few outputs and calling it evaluation - Single-metric gates: one aggregate number hiding per-slice regressions - LLM-judge-only: no deterministic floor, so the gate inherits the judge's noise - Threshold-on-the-fly: setting the cutoff after seeing results to make it pass - Framework-as-strategy: adopting one vendor tool as the whole eval program - Fine-tune-as-strategy: reaching for SFT/RFT/LoRA before proving the failure is learned behavior rather than prompt, context, tools, or product spec - Technique soup: stacking CoT, self-consistency, rerankers, judges, and post-training without isolating which intervention caused the lift ## Navigation Resources: - [references/dataset-construction.md](references/dataset-construction.md) - Sourcing questions, authoring ideal answers, run→compare→tune loop - [references/llm-judge-bias.md](references/llm-judge-bias.md) - Judge bias taxonomy and controls - [references/framework-integration.md](references/framework-integration.md) - Framework selection and integration snippets - [references/threshold-derivation.md](references/threshold-derivation.md) - Deriving thresholds and gates from labeled data - [references/flake-and-reproducibility.md](references/flake-and-reproducibility.md) - Flake, seeds, contamination, leakage - [references/eval-statistics.md](references/eval-statistics.md) - Bootstrap CIs, McNemar, power/MDE, FDR, variance reduction - [references/llm-optimization-technique-map.md](references/llm-optimization-technique-map.md) - Maximum-performance technique ladder and eval gates - [references/fine-tuning-eval-loop.md](references/fine-tuning-eval-loop.md) - Eval-first fine-tuning decisions, SFT/preference/RFT/PEFT selection, split hygiene, promotion gates - [references/online-production-eval.md](references/online-production-eval.md) - Offline-online correlation, A/B+guardrails, drift, replay, HITL - [references/safety-redteam-eval.md](references/safety-redteam-eval.md) - Refusal precision/recall, jailbreak/injection, harm rubrics, robustness - [references/advanced-judging.md](references/advanced-judging.md) - Juries, fine-tuned judges, scoring methods, calibration, agentic reward - [data/sources.json](data/sources.json) - Sources to verify against Related skills: - [ai-llm](../ai-llm/SKILL.md), [ai-rag](../ai-rag/SKILL.md), [ai-coding-agents-observability-evals](../ai-coding-agents-observability-evals/SKILL.md) ## Fact-Checking - Eval framework APIs (inspect-ai, lighteval, Ragas, DeepEval, promptfoo, Braintrust) change across releases. Verify current API and version against official docs before recommending a specific call or flag. - Fine-tuning platform support, model eligibility, dataset schemas, and RFT/grader APIs move quickly. Verify the current official docs before recommending a specific model, endpoint, hyperparameter, or CLI. - Optimization-method papers from arXiv are often preprint-only and benchmark-sensitive. Treat unreplicated methods as `validate`, not `promote`, until they beat a strong local baseline with cost/latency/safety gates. - Judge-bias findings (position, length, self-preference) are well-replicated through 2025-2026, but specific magnitudes are model- and prompt-dependent — re-measure on your own setup; do not quote a fixed number as universal. - Fine-tuned-judge models (Prometheus, JudgeLM, and successors) and jailbreak attack/defense results move fast — verify the current model, license, and benchmark-agreement claims before recommending a specific judge or asserting a model is robust to a given attack family. - Indirect prompt injection is the dominant agentic attack as of 2026; treat any "the agent is safe against injection" claim as requiring fresh adaptive testing. - Statistics methods (bootstrap, McNemar, FDR, power/MDE) are stable, but verify the exact API (scipy/statsmodels) before copying a call. - If web access is unavailable, mark framework-version claims as unverified. ## Learnings Loop Before applying this skill on a non-trivial task, read `learnings.consolidated.md` (and `learnings.md` if present). After applying it, append one dated bullet to `learnings.md` via `agents-skills-feedback-loop/scripts/append_learning.py` if you hit a pattern, mistake, or surprising fact. Do not modify `SKILL.md` itself.