Ac Model Evaluation

Building evaluation suites that decide between models for a task: what the eval measures before any prompt is written, constructing cases from real failure modes and coding them from task-native artifacts, contamination and why held-out data no longer guarantees freshness, automatic scoring with exact-match and judge limits, pass-at-k and aggregate metrics read honestly, human comparison and preference data as a separate instrument, and the written evaluation record. It designs the measurement; it never certifies a model as good and never recommends a vendor. Use when asking "model evaluation", "eval suite design", "build a benchmark", "contamination", "held-out set", "LLM as judge", "pass at k", "model comparison", "regression suite for model upgrades", "golden set", "cost of evaluation". Not for improving a prompt against fixed cases (ac-prompt-engineer), not for running the harness code itself (ac-llm-api-integration), not for evaluating datasets as research evidence (ac-alternative-data).

jiayaoqijia Updated 81 repo stars

File contents

jiayaoqijia/cryptoskill commit 01cbc3750d

Frequently asked questions

npx get-skill add jiayaoqijia/ac-model-evaluation