Run Evals

Run Evals

Run structured evaluations on an Agent Skill. Each eval prompt executes twice (with skill vs. without) in fresh context, outputs are graded against assertions, and a summary report is saved.

Prerequisites

• Target skill exists at ~/.cursor/skills/<skill-name>/SKILL.md
• Eval definitions exist at ~/.cursor/skills/<skill-name>/evals/evals.json
• See reference.md for evals.json schema and directory layout

Workflow

1. Ask which skill to evaluate

If the user did not name a skill, ask:

Which skill should I evaluate? (e.g. clean-notes, executive-email, google-tasks-prioritize)

Validate the skill exists:

~/.cursor/skills/<skill-name>/SKILL.md
~/.cursor/skills/<skill-name>/evals/evals.json

If either file is missing, stop and tell the user what to create.

2. Read eval definitions

Read ~/.cursor/skills/<skill-name>/evals/evals.json.

For each eval entry, note:

• id — eval identifier
• prompt — user message to run
• expected_output — human-readable success criteria
• files (optional) — input file paths relative to the skill directory
• assertions — verifiable pass/fail checks

3. Prepare results directories

Create or clear results directories under the target skill:

~/.cursor/skills/<skill-name>/evals/results/
├── with-skill/
├── without-skill/
└── summary.md          # written in step 6

Do not carry forward outputs from prior eval runs unless the user explicitly asks to append.

4. Run each eval twice in fresh context

For every eval entry, run the prompt twice — once with the skill, once without. Each run must use a fresh context so prior runs do not influence results.

Fresh context rule: Launch each run as an isolated subagent via the Task tool (subagent_type: generalPurpose). Do not execute multiple eval runs in the same conversation thread. Wait for each subagent to finish before starting the next.

4a. With-skill run

Launch a subagent with:

• Read and follow ~/.cursor/skills/<skill-name>/SKILL.md
• Attach any files listed in the eval's files array (resolve paths relative to ~/.cursor/skills/<skill-name>/)
• Execute the eval prompt exactly as written
• Return the full output (markdown/text and list of any files created)

Save the subagent's returned output to:

~/.cursor/skills/<skill-name>/evals/results/with-skill/eval-<id>.md

Prepend a header to the saved file:

# Eval <id> — With Skill

- Skill: <skill-name>
- Prompt: {prompt}
- Run: with-skill

---

{output}

4b. Without-skill run

Launch a new subagent with:

• Do not read or apply any skill file
• Attach the same input files (if any)
• Execute the same eval prompt
• Return the full output

Save to:

~/.cursor/skills/<skill-name>/evals/results/without-skill/eval-<id>.md

Use the same header format, with Run: without-skill.

Run order: Complete both runs for eval id N before starting eval id N+1. Order within an eval: with-skill first, then without-skill.

5. Grade each output

For each eval and each run (with-skill / without-skill), grade every assertion in evals.json.

Grading rules:

• Return PASS or FAIL for each assertion
• Include a one-sentence reason citing specific evidence from the saved output
• Do not give the benefit of the doubt — if evidence is missing or vague, grade FAIL
• Grade with-skill and without-skill outputs independently

Save per-eval grading alongside results:

~/.cursor/skills/<skill-name>/evals/results/with-skill/eval-<id>-grading.json
~/.cursor/skills/<skill-name>/evals/results/without-skill/eval-<id>-grading.json

Grading JSON format:

{
  "eval_id": 1,
  "run": "with-skill",
  "assertion_results": [
    {
      "text": "Assertion text from evals.json",
      "result": "PASS",
      "reason": "One-sentence evidence-based reason."
    }
  ]
}

Perform grading in fresh context when possible — use a subagent that receives only the saved output file, the assertions, and the expected output.

6. Write summary

Save ~/.cursor/skills/<skill-name>/evals/results/summary.md with:

# Eval Summary — {skill-name}

Run date: {ISO date}
Evals: {count}

## Results overview

| Eval ID | With skill | Without skill |
| --- | --- | --- |
| {id} | {passed}/{total} | {passed}/{total} |

## Passed in both runs

- {assertion text} (eval {id})

## Passed only with skill

- {assertion text} (eval {id}) — without-skill reason: {one sentence}

## Failed in both runs

- {assertion text} (eval {id}) — with-skill reason: {one sentence}; without-skill reason: {one sentence}

## Skill value

{1–3 sentences: where the skill clearly helped, where it did not, recommended next steps}

Classification rules:

• Passed in both — PASS in with-skill and without-skill grading
• Passed only with skill — PASS with-skill, FAIL without-skill
• Failed in both — FAIL in both runs

If an assertion passes without-skill but fails with-skill, note it separately under Regressions (unexpected — investigate skill instructions).

7. Confirm with the user

Report:

• Skill evaluated and eval count
• Path to summary.md
• Top-line: how many assertions passed only with skill vs. both vs. neither
• Obvious skill improvement opportunities (if any)

Quality rules

• Never skip the without-skill baseline run
• Never reuse conversation context between runs — always subagent isolation
• Never grade without reading the saved output files
• Never invent assertions not in evals.json
• If a run fails (tool error, timeout), record the failure in the output file and mark all assertions FAIL with reason "Run did not complete"

Example prompts

1. "Run evals on clean-notes"
2. "Evaluate the executive-email skill"
3. "Benchmark google-tasks-prioritize against its evals.json"

Additional resources

• reference.md — evals.json schema, directory layout, subagent prompt templates