# Agentbench Eval

> AgentBench evaluation harness for claudemem. Covers pre-indexed repos, experiment conditions, running benchmarks, analyzing results, and managing index archives. Use when working on eval infrastructure, running experiments, or interpreting benchmark results.

- Skill: `madappgang/agentbench-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add madappgang/agentbench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/madappgang/agentbench-eval/raw
- Safety review: PASS (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: MadAppGang (https://skillmd.com/u/madappgang)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/madappgang/agentbench-eval

---


# AgentBench Evaluation Skill

Run and manage claudemem evaluation experiments against the eth-sri/agentbench benchmark (138 instances, 12 Python repos).

## Repo Location

The agentbench repo is a sibling of the claudemem repo:
```
../agentbench/
```

All paths below are relative to the agentbench repo root unless noted otherwise.

## Data Layout

```
data/
├── eval-repos/              # 12 cloned repos with pre-built .claudemem/ indexes
│   └── {slug}/.claudemem/   # AST, symbols, vectors, enrichment per repo
├── archives/                # Immutable index snapshots (tar.gz)
│   ├── indexes-20260304-deepseek.tar.gz          # v2 full: 12 repos (1.2GB)
│   └── indexes-20260304-deepseek-11of12.tar.gz   # v2: 11 repos enriched (830MB)
├── eval-cache/              # Runtime sentinel cache
└── eval-generated/          # Generated CLAUDE.md files
```

## Index Specs (v2, 2026-03-04)

- Enrichment model: `deepseek/deepseek-v3.2` via OpenRouter
- Embedding model: `qwen/qwen3-embedding-8b` via OpenRouter
- Total: 12 repos, ~39K symbols, ~36K enrichment docs, ~1.9GB indexes
- Cost: ~$0.25 total via OpenRouter

### Repo Inventory

| Slug | Files | Symbols | Docs | Size |
|------|-------|---------|------|------|
| ansible_ansible | 1758 | 4,597 | 7,214 | 176M |
| getzep_graphiti | 115 | 434 | 635 | 43M |
| huggingface_smolagents | 70 | 475 | 621 | 20M |
| huggingface_transformers | 3082 | 0 | 0 | ~640M |
| jlowin_fastmcp | 417 | 1,914 | 3,348 | 92M |
| openai_openai-agents-python | 477 | 3,504 | 3,620 | 98M |
| opshin_opshin | 125 | 714 | 855 | 24M |
| pdm-project_pdm | 215 | 1,183 | 1,480 | 34M |
| qodo-ai_pr-agent | 114 | 328 | 835 | 21M |
| tinygrad_tinygrad | 884 | 20,018 | 6,345 | 400M |
| vibrantlabsai_ragas | 401 | 1,357 | 2,514 | 57M |
| wagtail_wagtail | 2270 | 4,434 | 8,419 | 261M |

Note: `huggingface_transformers` has 0 symbols/docs (tree-sitter WASM errors on metaprogramming). Vector search still works.

## Experiment Conditions

| Condition | Type | Workers | What it does |
|-----------|------|---------|--------------|
| `no_plan` | Baseline | 2 | Raw Claude Code, no AGENTS.md |
| `claudemem_full` | Per-instance | 2 | claudemem map+search → AGENTS.md per task |
| `dc_planner` | Cross-instance | 1 | Dynamic Cheatsheet — learns across tasks |
| `ace_planner` | Cross-instance | 1 | ACE reflector+curator playbook — learns across tasks |

### Instance Filter (24 instances, 2 per repo)

Hardcoded in `scripts/agentbench/run_harness/run_condition.py`.

## Common Tasks

### Run an Experiment

```bash
cd scripts/agentbench/run_harness
python run_condition.py <condition>
# e.g.: python run_condition.py no_plan
#        python run_condition.py claudemem_full
#        python run_condition.py dc_planner
#        python run_condition.py ace_planner
```

Never pass filter directly in shell — pipe `|` gets escaped by zsh.

### Restore Indexes (New Machine)

```bash
# From the agentbench repo root:
./scripts/agentbench/run_harness/restore_indexes.sh
# or specify archive:
./scripts/agentbench/run_harness/restore_indexes.sh \
  --archive data/archives/indexes-20260304-deepseek.tar.gz
```

### Re-Index a Single Repo

```bash
# Full enrichment (~$0.02/repo)
CLAUDEMEM_LLM=or/deepseek/deepseek-v3.2 claudemem index --force data/eval-repos/{slug}

# Fast index without enrichment (map works, search doesn't)
claudemem index --no-llm data/eval-repos/{slug}
```

### Create Archive Snapshot

```bash
cd data/eval-repos
tar czf ../archives/indexes-$(date +%Y%m%d)-deepseek.tar.gz */.claudemem/
```

### Check Experiment Results

```bash
# Results live at:
ls scripts/agentbench/run_harness/output/agentbench/eth-sri_agentbench/{condition}/

# Evaluate:
cd scripts/agentbench/run_harness
python evaluate.py --condition <condition> --run_id <N>

# Analyze:
python analyze.py
```

### DC/ACE Training Data

- DC cheatsheets: `plans/dynamic_cheatsheet/{model}/cheatsheet_{repo}.txt`
- ACE playbooks: `plans/ace_playbook/{model}/playbook_{repo}.json`
- History: `*_history/` subdirectories track evolution across instances

## Key Gotchas

1. **Always use `run_condition.py`** — shell escaping of `|` in filters breaks otherwise
2. **DC/ACE are sequential** (workers=1) — they learn across instances, can't parallelize
3. **Index cache is 3-level**: in-process set → `index.db` file → sentinel in `data/eval-cache/`
4. **Model keys are short names**: `sonnet-4-5` not `claude-sonnet-4-5-20250929`
5. **`generate.py` uses `fire.Fire(main)`** — both positional and `--flag=val` args work
6. **`--no-llm` indexes**: only give `map` (PageRank symbols). Full enrichment enables semantic `search`
7. **All paths are repo-relative** — no `~/.claudemem/` dependencies
8. **Enrichment model**: deepseek/deepseek-v3.2 was selected from 76 benchmark runs (composite score 0.886)

## Architecture (Key Files)

| File | Purpose |
|------|---------|
| `src/agentbench/planners/claudemem_planner.py` | ClaudememPlanner — indexes repo, generates AGENTS.md |
| `src/agentbench/planners/ace/ace.py` | ACE planner — reflector+curator playbook |
| `src/agentbench/planners/evo_reproducer/evo_reproducer.py` | DC/EvoReproducer planner — dynamic cheatsheet |
| `scripts/agentbench/run_harness/generate.py` | Main harness entry point |
| `scripts/agentbench/run_harness/run_condition.py` | Launch helper (handles filter escaping) |
| `scripts/agentbench/run_harness/restore_indexes.sh` | Archive restore script |
| `scripts/agentbench/run_harness/evaluate.py` | Result evaluator |
| `scripts/agentbench/run_harness/analyze.py` | Cross-condition analyzer |