Agentbench Eval

AgentBench Evaluation Skill

Run and manage claudemem evaluation experiments against the eth-sri/agentbench benchmark (138 instances, 12 Python repos).

Repo Location

The agentbench repo is a sibling of the claudemem repo:

../agentbench/

All paths below are relative to the agentbench repo root unless noted otherwise.

Data Layout

data/
├── eval-repos/              # 12 cloned repos with pre-built .claudemem/ indexes
│   └── {slug}/.claudemem/   # AST, symbols, vectors, enrichment per repo
├── archives/                # Immutable index snapshots (tar.gz)
│   ├── indexes-20260304-deepseek.tar.gz          # v2 full: 12 repos (1.2GB)
│   └── indexes-20260304-deepseek-11of12.tar.gz   # v2: 11 repos enriched (830MB)
├── eval-cache/              # Runtime sentinel cache
└── eval-generated/          # Generated CLAUDE.md files

Index Specs (v2, 2026-03-04)

• Enrichment model: deepseek/deepseek-v3.2 via OpenRouter
• Embedding model: qwen/qwen3-embedding-8b via OpenRouter
• Total: 12 repos, ~39K symbols, ~36K enrichment docs, ~1.9GB indexes
• Cost: ~$0.25 total via OpenRouter

Repo Inventory

Note: huggingface_transformers has 0 symbols/docs (tree-sitter WASM errors on metaprogramming). Vector search still works.

Experiment Conditions

Instance Filter (24 instances, 2 per repo)

Hardcoded in scripts/agentbench/run_harness/run_condition.py.

Common Tasks

Run an Experiment

cd scripts/agentbench/run_harness
python run_condition.py <condition>
# e.g.: python run_condition.py no_plan
#        python run_condition.py claudemem_full
#        python run_condition.py dc_planner
#        python run_condition.py ace_planner

Never pass filter directly in shell — pipe | gets escaped by zsh.

Restore Indexes (New Machine)

# From the agentbench repo root:
./scripts/agentbench/run_harness/restore_indexes.sh
# or specify archive:
./scripts/agentbench/run_harness/restore_indexes.sh \
  --archive data/archives/indexes-20260304-deepseek.tar.gz

Re-Index a Single Repo

# Full enrichment (~$0.02/repo)
CLAUDEMEM_LLM=or/deepseek/deepseek-v3.2 claudemem index --force data/eval-repos/{slug}

# Fast index without enrichment (map works, search doesn't)
claudemem index --no-llm data/eval-repos/{slug}

Create Archive Snapshot

cd data/eval-repos
tar czf ../archives/indexes-$(date +%Y%m%d)-deepseek.tar.gz */.claudemem/

Check Experiment Results

# Results live at:
ls scripts/agentbench/run_harness/output/agentbench/eth-sri_agentbench/{condition}/

# Evaluate:
cd scripts/agentbench/run_harness
python evaluate.py --condition <condition> --run_id <N>

# Analyze:
python analyze.py

DC/ACE Training Data

• DC cheatsheets: plans/dynamic_cheatsheet/{model}/cheatsheet_{repo}.txt
• ACE playbooks: plans/ace_playbook/{model}/playbook_{repo}.json
• History: *_history/ subdirectories track evolution across instances

Key Gotchas

1. Always use run_condition.py — shell escaping of | in filters breaks otherwise
2. DC/ACE are sequential (workers=1) — they learn across instances, can't parallelize
3. Index cache is 3-level: in-process set → index.db file → sentinel in data/eval-cache/
4. Model keys are short names: sonnet-4-5 not claude-sonnet-4-5-20250929
5. generate.py uses fire.Fire(main) — both positional and --flag=val args work
6. --no-llm indexes: only give map (PageRank symbols). Full enrichment enables semantic search
7. All paths are repo-relative — no ~/.claudemem/ dependencies
8. Enrichment model: deepseek/deepseek-v3.2 was selected from 76 benchmark runs (composite score 0.886)

Architecture (Key Files)