AI Evals Design

Operational playbook for designing production LLM evaluation in 2026. Covers eval framework selection (Promptfoo, DeepEval, Maxim, LangSmith, Braintrust, Phoenix, Galileo, Patronus), golden dataset construction (size, stratification, versioning), synthetic data generation, LLM-as-judge design with G-Eval methodology (Liu et al., EMNLP 2023), preference-leakage mitigation (Wong et al. 2025), statistical significance testing (McNemar's, paired t, Bayesian pairwise, bootstrap), RAG metrics (RAGAS, TruLens RAG Triad), agent trajectory evaluation, drift detection, eval-driven development (Red Hat, March 2026), CI/CD prompt-regression gates, and benchmark literacy (GAIA, BFCL, SWE-Bench Verified, MMLU-Pro, GPQA Diamond, HELM, τ-Bench). Use when designing an eval suite, validating a prompt change, gating a deploy, or diagnosing production quality drift. Output: eval verdicts with statistical evidence, not vibe judgments.

Muvon Updated 6 repo stars

File contents

Muvon/octomind-tap commit 93abac38b0

Frequently asked questions

npx get-skill add muvon/ai-evals-design