How do I install the Run LLM Evals skill?

Run npx skillmds@latest add hiteshbandhu/run-llm-evals in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Run LLM Evals safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Run LLM Evals?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Run LLM Evals free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Run LLM Evals?

hiteshbandhu (@hiteshbandhu) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Run LLM Evals by hiteshbandhu — Agent Skill

Name: Run LLM Evals
Author: hiteshbandhu

Run LLM Evals

Runs checklists and workflows for designing, operating, and improving LLM evals — domain harnesses, agent/voice evals, LLM judges, metrics workshops, product flywheels, dynamic search eval, enterprise conversation intelligence. Use when the user says "evals", "LLM judge", "eval harness", "regression on prompts", "agent eval", "eval dataset", or needs to ship with measurement before prod.

hiteshbandhu Updated Sep 17, 2026

File contents

Run LLM evals

Action playbook from twenty-five AI Engineer eval/benchmark talks. Do not summarize talks — pick a workflow and execute it.

Supporting files (read when needed):

workflows.md — workflows A–N (steps, deliverables, stop conditions)
source-index.md — src-NNN → talk learnings in ingest-into-skills

Optional: {SKILL_OUTPUT_DIR}/run-llm-evals/

Step 0 — Pick workflow

Use the decision tree below. Open the matching section in workflows.md.

What is the user trying to do?
├─ Start or justify an eval program                    → A
├─ Build domain-specific eval from scratch             → B
├─ Design product/LLM app eval (not unit tests)        → C
├─ Define metrics/rubrics that match humans            → D
├─ Braintrust-style playground → CI → online eval      → E
├─ Ship agentic apps with trace-native eval            → F
├─ Voice / phone agent simulation eval                 → G
├─ Close observability ↔ eval gap (Azure agents)       → H
├─ Calibrate LLM judges / adversarial fuzzing          → I
├─ Benchmark models (perf + knowledge + honesty)       → J
├─ Enterprise CX: 100% conversations + labeling        → K
├─ Zapier-style PM+eng eval ops & post-ship feedback   → L
├─ Evaluate web/RAG search APIs (dynamic sets)         → M
└─ Subjective media eval or long-lived architecture    → N

Stop summarizing once a workflow is identified — run its checklist.

Install

Copy this folder into your agent’s skills directory:

cp -r skills/run-llm-evals ~/.claude/skills/
cp -r skills/run-llm-evals ~/.cursor/skills/
cp -r skills/run-llm-evals ~/.codex/skills/

From skills-i-use or ingest-into-skills (playlists/evals-benchmarks-ai-engineer/).

Cross-cutting rules

Rule	Source
Evals ≠ unit tests; cover distributions and offense, not one-shot demos	[src-005 @ 2:57]
Look at data before fancy judges	[src-006 @ 7:15], [src-009 @ 8:25]
Align offline scorers with online logging	[src-001 @ 3:23]
Layer evals on routers, tools, and final answers	[src-007 @ 4:46]
Calibrate LLM judges; meta-eval the judge	[src-013 @ 8:15]
Separate core-traffic vs aspirational datasets	[src-020 @ 15:03]

Disputed steps: see source-index.md. Name workflow A–N; save artifacts to ./skill-outputs/run-llm-evals/ when requested; do not auto-commit.

---name: run-llm-evalsdescription: Runs checklists and workflows for designing, operating, and improving LLM evals — domain harnesses, agent/voice evals, LLM judges, metrics workshops, product flywheels, dynamic search eval, enterprise conversation intelligence. Use when the user says "evals", "LLM judge", "eval harness", "regression on prompts", "agent eval", "eval dataset", or needs to ship with measurement before prod.---# Run LLM evalsAction playbook from twenty-five AI Engineer eval/benchmark talks. **Do not summarize talks** — pick a workflow and execute it.**Supporting files** (read when needed):- [workflows.md](workflows.md) — workflows A–N (steps, deliverables, stop conditions)- [source-index.md](source-index.md) — src-NNN → talk learnings in ingest-into-skillsOptional: `{SKILL_OUTPUT_DIR}/run-llm-evals/`---## Step 0 — Pick workflowUse the decision tree below. Open the matching section in [workflows.md](workflows.md).```What is the user trying to do?├─ Start or justify an eval program → A├─ Build domain-specific eval from scratch → B├─ Design product/LLM app eval (not unit tests) → C├─ Define metrics/rubrics that match humans → D├─ Braintrust-style playground → CI → online eval → E├─ Ship agentic apps with trace-native eval → F├─ Voice / phone agent simulation eval → G├─ Close observability ↔ eval gap (Azure agents) → H├─ Calibrate LLM judges / adversarial fuzzing → I├─ Benchmark models (perf + knowledge + honesty) → J├─ Enterprise CX: 100% conversations + labeling → K├─ Zapier-style PM+eng eval ops & post-ship feedback → L├─ Evaluate web/RAG search APIs (dynamic sets) → M└─ Subjective media eval or long-lived architecture → N```**Stop summarizing** once a workflow is identified — run its checklist.---## InstallCopy this folder into your agent’s skills directory:```bashcp -r skills/run-llm-evals ~/.claude/skills/cp -r skills/run-llm-evals ~/.cursor/skills/cp -r skills/run-llm-evals ~/.codex/skills/```From [skills-i-use](https://github.com/hiteshbandhu/skills-i-use) or ingest-into-skills (`playlists/evals-benchmarks-ai-engineer/`).---## Cross-cutting rules| Rule | Source ||------|--------|| Evals ≠ unit tests; cover distributions and offense, not one-shot demos | [src-005 @ 2:57] || Look at data before fancy judges | [src-006 @ 7:15], [src-009 @ 8:25] || Align offline scorers with online logging | [src-001 @ 3:23] || Layer evals on routers, tools, and final answers | [src-007 @ 4:46] || Calibrate LLM judges; meta-eval the judge | [src-013 @ 8:15] || Separate core-traffic vs aspirational datasets | [src-020 @ 15:03] |Disputed steps: see [source-index.md](source-index.md). Name workflow A–N; save artifacts to `./skill-outputs/run-llm-evals/` when requested; do not auto-commit.

Frequently asked questions