# Run Evals > Runs structured evaluations on a Cursor Agent Skill by executing each eval prompt with and without the skill, grading assertions, and writing a summary report. Use when the user asks to run evals, evaluate a skill, test skill quality, or benchmark skill performance. - Skill: `cgray-redhat/run-evals` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add cgray-redhat/run-evals` - Raw SKILL.md: https://api.skillmd.com/api/skills/cgray-redhat/run-evals/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: cgray-redhat (https://skillmd.com/u/cgray-redhat) - Updated: 2026-09-22 - Page: https://skillmd.com/skills/cgray-redhat/run-evals --- # Run Evals Run structured evaluations on an Agent Skill. Each eval prompt executes twice (with skill vs. without) in **fresh context**, outputs are graded against assertions, and a summary report is saved. ## Prerequisites - Target skill exists at `~/.cursor/skills//SKILL.md` - Eval definitions exist at `~/.cursor/skills//evals/evals.json` - See [reference.md](reference.md) for `evals.json` schema and directory layout ## Workflow ### 1. Ask which skill to evaluate If the user did not name a skill, ask: > Which skill should I evaluate? (e.g. `clean-notes`, `executive-email`, `google-tasks-prioritize`) Validate the skill exists: ``` ~/.cursor/skills//SKILL.md ~/.cursor/skills//evals/evals.json ``` If either file is missing, stop and tell the user what to create. ### 2. Read eval definitions Read `~/.cursor/skills//evals/evals.json`. For each eval entry, note: - `id` — eval identifier - `prompt` — user message to run - `expected_output` — human-readable success criteria - `files` (optional) — input file paths relative to the skill directory - `assertions` — verifiable pass/fail checks ### 3. Prepare results directories Create or clear results directories under the target skill: ``` ~/.cursor/skills//evals/results/ ├── with-skill/ ├── without-skill/ └── summary.md # written in step 6 ``` Do not carry forward outputs from prior eval runs unless the user explicitly asks to append. ### 4. Run each eval twice in fresh context For **every** eval entry, run the prompt **twice** — once with the skill, once without. Each run must use a **fresh context** so prior runs do not influence results. **Fresh context rule:** Launch each run as an isolated subagent via the Task tool (`subagent_type: generalPurpose`). Do not execute multiple eval runs in the same conversation thread. Wait for each subagent to finish before starting the next. #### 4a. With-skill run Launch a subagent with: - Read and follow `~/.cursor/skills//SKILL.md` - Attach any files listed in the eval's `files` array (resolve paths relative to `~/.cursor/skills//`) - Execute the eval `prompt` exactly as written - Return the full output (markdown/text and list of any files created) Save the subagent's returned output to: ``` ~/.cursor/skills//evals/results/with-skill/eval-.md ``` Prepend a header to the saved file: ```markdown # Eval — With Skill - Skill: - Prompt: {prompt} - Run: with-skill --- {output} ``` #### 4b. Without-skill run Launch a **new** subagent with: - Do **not** read or apply any skill file - Attach the same input files (if any) - Execute the same eval `prompt` - Return the full output Save to: ``` ~/.cursor/skills//evals/results/without-skill/eval-.md ``` Use the same header format, with `Run: without-skill`. **Run order:** Complete both runs for eval `id` N before starting eval `id` N+1. Order within an eval: with-skill first, then without-skill. ### 5. Grade each output For each eval and each run (with-skill / without-skill), grade **every assertion** in `evals.json`. Grading rules: - Return **PASS** or **FAIL** for each assertion - Include a **one-sentence reason** citing specific evidence from the saved output - Do not give the benefit of the doubt — if evidence is missing or vague, grade FAIL - Grade with-skill and without-skill outputs independently Save per-eval grading alongside results: ``` ~/.cursor/skills//evals/results/with-skill/eval--grading.json ~/.cursor/skills//evals/results/without-skill/eval--grading.json ``` Grading JSON format: ```json { "eval_id": 1, "run": "with-skill", "assertion_results": [ { "text": "Assertion text from evals.json", "result": "PASS", "reason": "One-sentence evidence-based reason." } ] } ``` Perform grading in fresh context when possible — use a subagent that receives only the saved output file, the assertions, and the expected output. ### 6. Write summary Save `~/.cursor/skills//evals/results/summary.md` with: ```markdown # Eval Summary — {skill-name} Run date: {ISO date} Evals: {count} ## Results overview | Eval ID | With skill | Without skill | | --- | --- | --- | | {id} | {passed}/{total} | {passed}/{total} | ## Passed in both runs - {assertion text} (eval {id}) ## Passed only with skill - {assertion text} (eval {id}) — without-skill reason: {one sentence} ## Failed in both runs - {assertion text} (eval {id}) — with-skill reason: {one sentence}; without-skill reason: {one sentence} ## Skill value {1–3 sentences: where the skill clearly helped, where it did not, recommended next steps} ``` Classification rules: - **Passed in both** — PASS in with-skill and without-skill grading - **Passed only with skill** — PASS with-skill, FAIL without-skill - **Failed in both** — FAIL in both runs If an assertion passes without-skill but fails with-skill, note it separately under **Regressions** (unexpected — investigate skill instructions). ### 7. Confirm with the user Report: - Skill evaluated and eval count - Path to `summary.md` - Top-line: how many assertions passed only with skill vs. both vs. neither - Obvious skill improvement opportunities (if any) ## Quality rules - Never skip the without-skill baseline run - Never reuse conversation context between runs — always subagent isolation - Never grade without reading the saved output files - Never invent assertions not in `evals.json` - If a run fails (tool error, timeout), record the failure in the output file and mark all assertions FAIL with reason "Run did not complete" ## Example prompts 1. "Run evals on clean-notes" 2. "Evaluate the executive-email skill" 3. "Benchmark google-tasks-prioritize against its evals.json" ## Additional resources - [reference.md](reference.md) — evals.json schema, directory layout, subagent prompt templates