How do I install the Agent Eval skill?

Run npx skillmds@latest add tomevault-io/agent-eval-3 in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Agent Eval safe to use?

This skill has not completed SkillMD's automated safety review yet. Independent scanners report: SkillSpector: PASS, Skill Scanner: PASS. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Agent Eval?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Agent Eval free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Agent Eval?

tomevault-io (@tomevault-io) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Agent Eval by tomevault-io — Agent Skill

Name: Agent Eval
Author: tomevault-io

Agent Eval

Benchmark CodeGraph retrieval quality on a real codebase by comparing agent behavior with vs without CodeGraph. Use when the user runs /agent-eval or asks to test, benchmark, audit, or validate a codegraph version (the local dev build or a published npm version) against a language's repo. Use when this capability is needed.

tomevault-io Updated Sep 17, 2026

File contents

CodeGraph Quality Audit

Measures how much CodeGraph helps an agent versus plain grep/read, for a chosen codegraph version on a chosen real-world repo. Drives the harness in scripts/agent-eval/.

Prerequisites

tmux 3+, a logged-in claude CLI, node, git (macOS/Linux).
Run from the codegraph repo root.

Workflow

Copy this checklist:

- [ ] 1. Pick version (local or npm)
- [ ] 2. Pick language
- [ ] 3. Pick repo by size
- [ ] 4. Pick harness (headless / tmux / both)
- [ ] 5. Run audit.sh in the background
- [ ] 6. Report results

Step 1 — version. Ask with AskUserQuestion: which codegraph version to test. Offer "Local dev build" and "Latest published"; the free-text "Other" lets the user type a specific version (e.g. 0.7.10). Map the answer to a VERSION token:

"Local dev build" → local
"Latest published" → latest
a typed version → that string (e.g. 0.7.10)

Step 2 — language. Read .claude/skills/agent-eval/corpus.json. Ask with AskUserQuestion which language to test, listing the languages that have entries.

Step 3 — repo. From the chosen language's entries, ask which repo. Label each option with its size and file count, e.g. excalidraw — Medium (~600 files). Each entry carries the repo URL and a representative question.

Step 4 — harness. Ask with AskUserQuestion which harness to run, and map the answer to a MODE token:

"Headless" → headless — claude -p with stream-json: exact tokens/cost and a clean tool sequence (2 runs, fast, no TTY).
"Interactive (tmux)" → tmux — drives the real Claude TUI in tmux: faithful Explore-subagent behavior, metrics from session logs (2 runs, slower).
"Both" → all — headless + interactive (4 runs).

Step 5 — run. Launch in the background (sets the version, clones if missing, wipes + re-indexes, runs the chosen arms — several minutes):

scripts/agent-eval/audit.sh <VERSION> <repo-name> <repo-url> "<question>" <MODE>

Step 6 — report. When the job finishes, read the log and report per arm:

Headless (parse-run.mjs): total tool calls, file Reads, Grep/Bash, codegraph-tool calls, duration, total cost.
Interactive (parse-session.mjs): the VERDICT: codegraph_explore used Nx | Read N | Grep/Bash N and TOKENS: lines.

Lead with cost + tool/Read counts — they are the reliable signals; raw token in/out are confounded by subagent delegation and prompt caching. State whether codegraph reduced effort and whether both arms reached a correct answer.

Notes

The index is rebuilt every run (audit.sh wipes .codegraph) — different versions extract differently, so an index must be served by the same binary that built it.
audit.sh temporarily mutates the global codegraph install for the test, then restores your dev link via local-install.sh.
Corpus repos are cloned to /tmp/codegraph-corpus (reused if already present).
Add or edit repos in corpus.json (fields: name, repo, size, files, question).

Source: colbymchenry/codegraph — distributed by TomeVault.

---name: agent-eval-3description: Benchmark CodeGraph retrieval quality on a real codebase by comparing agent behavior with vs without CodeGraph. Use when the user runs /agent-eval or asks to test, benchmark, audit, or validate a codegraph version (the local dev build or a published npm version) against a language's repo. Use when this capability is needed.---# CodeGraph Quality AuditMeasures how much CodeGraph helps an agent versus plain grep/read, for a chosencodegraph version on a chosen real-world repo. Drives the harness in`scripts/agent-eval/`.## Prerequisites- `tmux` 3+, a logged-in `claude` CLI, `node`, `git` (macOS/Linux).- Run from the codegraph repo root.## WorkflowCopy this checklist:```- [ ] 1. Pick version (local or npm)- [ ] 2. Pick language- [ ] 3. Pick repo by size- [ ] 4. Pick harness (headless / tmux / both)- [ ] 5. Run audit.sh in the background- [ ] 6. Report results```**Step 1 — version.** Ask with `AskUserQuestion`: which codegraph version to test.Offer "Local dev build" and "Latest published"; the free-text "Other" lets theuser type a specific version (e.g. `0.7.10`). Map the answer to a VERSION token:- "Local dev build" → `local`- "Latest published" → `latest`- a typed version → that string (e.g. `0.7.10`)**Step 2 — language.** Read `.claude/skills/agent-eval/corpus.json`. Ask with`AskUserQuestion` which language to test, listing the languages that have entries.**Step 3 — repo.** From the chosen language's entries, ask which repo. Label eachoption with its size and file count, e.g. `excalidraw — Medium (~600 files)`.Each entry carries the `repo` URL and a representative `question`.**Step 4 — harness.** Ask with `AskUserQuestion` which harness to run, and mapthe answer to a MODE token:- "Headless" → `headless` — `claude -p` with stream-json: exact tokens/cost and a clean tool sequence (2 runs, fast, no TTY).- "Interactive (tmux)" → `tmux` — drives the real Claude TUI in tmux: faithful Explore-subagent behavior, metrics from session logs (2 runs, slower).- "Both" → `all` — headless + interactive (4 runs).**Step 5 — run.** Launch in the background (sets the version, clones if missing,wipes + re-indexes, runs the chosen arms — several minutes):```bashscripts/agent-eval/audit.sh <VERSION> <repo-name> <repo-url> "<question>" <MODE>```**Step 6 — report.** When the job finishes, read the log and report per arm:- Headless (`parse-run.mjs`): total tool calls, file `Read`s, Grep/Bash, codegraph-tool calls, duration, **total cost**.- Interactive (`parse-session.mjs`): the `VERDICT: codegraph_explore used Nx | Read N | Grep/Bash N` and `TOKENS:` lines.Lead with cost + tool/Read counts — they are the reliable signals; raw tokenin/out are confounded by subagent delegation and prompt caching. State whethercodegraph reduced effort and whether both arms reached a correct answer.## Notes- The index is rebuilt every run (`audit.sh` wipes `.codegraph`) — different versions extract differently, so an index must be served by the same binary that built it.- `audit.sh` temporarily mutates the global `codegraph` install for the test, then restores your dev link via `local-install.sh`.- Corpus repos are cloned to `/tmp/codegraph-corpus` (reused if already present).- Add or edit repos in `corpus.json` (fields: `name`, `repo`, `size`, `files`, `question`).---> Source: [colbymchenry/codegraph](https://github.com/colbymchenry/codegraph) — distributed by [TomeVault](https://tomevault.io).

Frequently asked questions