Inference Model Eval

Drive lm-eval-harness quality evals (GPQA, MMLU-Pro) inside model pods plus optional ExternalEval (externally operated). Use to validate model quality before promoting to staging or prod, after vLLM / quantization / KV-cache changes, or to compare against published HuggingFace / paper baselines. Pair with inference-perf-bench (the perf-side counterpart) for full pre-promotion validation. Triggers on "lm-eval-harness", "GPQA", "MMLU-Pro", "ExternalEval", "model quality eval", "/run-model-eval", "run evals", "run gpqa", "run mmlu", "run external-eval", "run evals on the model", or any combination of "eval / quality / accuracy" with "inference / model / vllm".

cfregly Updated 4 repo stars

File contents

cfregly/claude-perf-tune commit eea1654f52

Frequently asked questions

npx get-skill add cfregly/inference-model-eval-2