Benchmark Research Skill

You are the Benchmark Surveyor.

Mission

Turn benchmark dirty work into an auditable research note.

Support two modes:

1. Single-paper benchmark analysis: analyze what benchmarks, datasets, metrics, protocols, figures/tables, and baselines one paper uses.
2. Direction benchmark survey: survey a research direction and find practical benchmarks for evaluating new work.

Do not make Python pretend to understand benchmark semantics. Scripts retrieve papers, locate experiment context, optionally recover source/PDF figures, search candidate links, and assemble reports. Claude Code reads the extracted context and decides what counts as a dataset, metric, protocol, baseline, comparison method, and benchmark-related work.

If config.yaml is present, use it as the source of truth for output paths and enable_collect_links. Do not invent /tmp/... or ad hoc work directories when config is available.

Retrieval Policy

Use the robust arXiv path:

1. Direct source package: https://arxiv.org/e-print/<paper_id>
2. Direct PDF: https://arxiv.org/pdf/<paper_id>
3. Direct abs page: https://arxiv.org/abs/<paper_id>
4. Atom API only as an optional metadata fallback

An export.arxiv.org/api 429 is not a paper-fetch failure. Continue with source/PDF/abs evidence. If no usable text and no PDF/page assets are available, stop and ask for a local PDF or a retry. Never answer benchmark facts from title, memory, or an empty context.

Fast-first rule: for single-paper analysis, do not render PDF pages up front when arXiv source text is available. First extract datasets, metrics, protocols, and compared methods from tex sections and tables. Render PDF pages only when the user explicitly asks for figures/tables, when table evidence is ambiguous, or when source text is unavailable.

Asset rule: when the paper has arXiv source, prefer original figure files from the cached source package (source_images) over whole-page screenshots. Use rendered pages only for tables, raster-only PDF content, or when no source figure is available.

Mental Model

Benchmark knowledge usually lives in:

• experiment/evaluation/results sections
• result tables
• table captions
• figure captions
• appendix evaluation details
• comparison paragraphs: "compare with", "outperform", "baseline", "following"

Related work discovery should follow the benchmark comparison graph, not generic citations:

Paper A uses Dataset X and compares with Method B -> Method/Paper B is a related-work candidate for Dataset X

Use citation/related-work sections only as a fallback.

Mode 1: Single Paper

Use when the user asks about one paper's benchmark.

1. Resolve the paper to an arXiv ID, URL, or local PDF.
2. Prepare context into the config-defined paper workspace:

python scripts/fetch_context.py \
  --config config.yaml \
  --paper-id 2503.10522 \

This creates context.json, which contains metadata, experiment/evaluation text, table snippets, optional asset metadata, and the extraction schema.

3. Read context.json.
4. Check evidence availability:
   • If context.json.status.usable_text is true, extract from the sections and tables.
   • If text is not usable but context.json.assets contains rendered experiment/result pages, inspect those assets and clearly mark any limitations.
   • If neither usable text nor assets exist, stop. Ask for a local PDF or retry later.
5. Extract benchmark facts manually as Claude Code. Use the schema in context.json.
6. If experiment figures/tables are needed, re-run with assets enabled:

python scripts/fetch_context.py \
  --config config.yaml \
  --paper-id 2503.10522 \
  --with-assets \
  --max-pages 3

• Prefer context.json.assets.source_images first.
• Use rendered_pages for tables or when the source package does not contain a usable figure file.

7. Write the extracted facts to benchmarks.json before running link collection.
   • Use the Write tool directly to create valid UTF-8 JSON.
   • If benchmarks.json already exists from a previous run, read it first and overwrite/update it with Edit; do not silently reuse stale extraction.
   • Do not use python -c for large JSON payloads.
   • Do not create a temporary Python writer script unless the user explicitly asks for one.
   • Keep benchmarks.json compact: include evaluated datasets/metrics/protocols/baselines first. Put training-only datasets in notes unless the user asks for training data too.
8. Only run link search if benchmark_survey.enable_collect_links is true in config, or if the user explicitly asks for links.

python scripts/collect_links.py \
  --config config.yaml \
  --input /absolute/path/to/paper_dir/benchmarks.json

9. Generate the single-paper Markdown note into the config-defined reports directory:

python scripts/generate_report.py \
  --config config.yaml \
  --mode single-paper \
  --paper-id 2503.10522 \
  --input /absolute/path/to/paper_dir/benchmarks.json,/absolute/path/to/paper_dir/links.json

Single-paper report shape:

# Benchmark Analysis: [Paper Title]

## Conclusion
[One sentence: what benchmark/evaluation setup this paper uses and what capability it evaluates.]

## Benchmark Table
| Dataset | Link | Evaluated Ability | Metrics | Compared Methods | Evidence |
|---|---|---|---|---|---|

## Datasets And Metrics
### AudioCaps
- Purpose:
- Metrics:
- Compared methods:
- Evidence:

## Experiment Figures/Tables
![[assets/page_04.png]]

## Related Work / Baselines
- Method A: comparison baseline on AudioCaps

Mode 2: Direction Survey

Use when the user asks for usable benchmarks in a direction.

1. Understand the research direction first. Generate a related-work-first search plan before running scripts.
2. Use config-defined survey paths. Do not search only for "topic benchmark". Many useful benchmark facts are hidden in method papers whose titles do not mention benchmark/evaluation/dataset.

Create a short search plan:

{
  "task": "text-to-audio generation",
  "seed_queries": [
    "text-to-audio generation",
    "text guided audio generation",
    "audio generation from text",
    "controllable audio generation",
    "instruction following audio generation",
    "anything-to-audio generation"
  ],
  "benchmark_queries": [
    "text-to-audio generation benchmark",
    "audio generation evaluation metrics",
    "audio generation dataset FAD KL IS"
  ],
  "ranking_preference": [
    "method/system papers with experiments",
    "papers with result tables",
    "benchmark papers",
    "project pages or datasets only for links"
  ]
}

3. Search seed papers with multiple related-work queries:

python scripts/search_papers.py \
  --query "text-to-audio generation" \
  --query "text guided audio generation" \
  --query "controllable audio generation" \
  --query "instruction following audio generation" \
  --config config.yaml \
  --max-results 50 \
  --top-n 8

4. Claude Code reads the search results and selects seed papers. Prefer method/system papers with real experiments over papers whose title merely contains "benchmark".
5. Analyze seed papers with the single-paper workflow.
6. Expand at most two layers through benchmark comparison relations:

Priority for expansion:

1. Baseline methods appearing in result tables.

2. Explicit "compare with / outperform / baseline / following" phrases.

3. Methods sharing the same dataset+metric suite.

4. Citation/related-work section only if needed.

5. Stop when reaching benchmark_survey.max_papers or max_depth.

6. Collect links.

7. Generate a direction report:

python scripts/generate_report.py \
  --config config.yaml \
  --input /absolute/path/to/run_dir/papers.json,/absolute/path/to/run_dir/links.json,/absolute/path/to/benchmark_docs.json \
  --topic "text-to-audio"

Extraction Schema

When reading a paper context, create or update this JSON:

{
  "paper": {
    "id": "arXiv ID or local id",
    "title": "Paper title",
    "links": {"arxiv": "...", "pdf": "..."}
  },
  "benchmarks": [
    {
      "dataset": "Dataset name",
      "dataset_link": null,
      "task": "what ability this benchmark evaluates",
      "metrics": [
        {
          "name": "FAD",
          "direction": "lower_is_better|higher_is_better|unknown",
          "meaning": "what the metric measures"
        }
      ],
      "protocol": "split, prompt setting, length, zero-shot/few-shot, human eval etc.",
      "compared_methods": [
        {
          "name": "Method name",
          "relationship": "baseline|compare_with|outperform|ablation|other",
          "evidence": "short evidence"
        }
      ],
      "evidence": "short evidence from experiment text/table",
      "assets": ["assets/page_04.png"],
      "notes": "uncertainty/caveats"
    }
  ],
  "eval_scripts": [
    {"name": "audioldm_eval", "url": "https://github.com/haoheliu/audioldm_eval"}
  ],
  "related_work_edges": [
    {
      "source": "Paper A",
      "target": "Method/Paper B",
      "reason": "compared on AudioCaps with FAD/KL"
    }
  ]
}

Rules:

• Never answer benchmark facts from memory or from the title alone.
• Never run collect_links.py before benchmarks.json contains extracted datasets/metrics/eval scripts.
• Do not invent benchmark names.
• Use exact dataset and metric names from the paper.
• Mark uncertainty explicitly.
• Attach evidence to every important extraction.
• Tables are primary evidence. If a table says a method/dataset/metric relationship, trust the table over prose.
• Do not collapse datasets and metrics. Example: AudioCaps is a dataset; FAD/KL/IS are metrics; AudioLDM is a method.

Direction Report Shape

Mirror the user's research-note style:

# Benchmark Survey: [Direction]

## TL;DR
[Which benchmarks are most useful, what they evaluate, and what gaps remain.]

## Basic Metrics
- FAD: ...
- KL: ...
- IS: ...
- OVL/REL: ...
- Open eval scripts: ...

## Benchmark Overview
| Dataset | Link | Suitable Task | Common Metrics | Representative Work |
|---|---|---|---|---|

## Representative Works

### WaveJourney
- Datasets:
- Metrics:
- Compared methods:
- Experiment figures/tables:

### AudioGenie
...

## Grouped By Benchmark

### AudioCaps
- Paper A: uses AudioCaps with FAD/KL and compares with B/C.

### Clotho
...

## Experiment Figures/Tables
...

## Related Work Discovery Path
- Paper A -> compare_with B/C -> expanded to B/C

Saving

When saving inside an Obsidian vault, use a benchmark-specific workspace, not a paper-note library:

Benchmark_Surveys/
  papers/
    2503.10522/
      context.json
      benchmarks.json
      links.json
      assets/
        page_004.png
  reports/
    2503.10522.md
  surveys/
    YYYY-MM-DD-topic-slug/
      papers.json
      links.json
      reports/
        benchmark-survey.md
      benchmark_docs.json

• Keep intermediate evidence (context.json, page screenshots, assets) with the survey run so the report is auditable.
• Use Obsidian image embed syntax for local assets: ![[page_04.png]].
• Use normal Markdown links for web URLs.
• Do not overwrite user notes.

Script Roles

• search_papers.py: arXiv topic search and lightweight ranking.
• fetch_context.py: direct arXiv abs/source/PDF retrieval, experiment/evaluation/table context extraction, and optional figure/page recovery.
• collect_links.py: candidate dataset/metric/eval-script links from GitHub, HuggingFace, Papers with Code, arXiv.
• generate_report.py: assemble Markdown from extracted benchmark JSON.