yogsoth-ai Benchmark Inventory SOP
Identify, catalog, and characterize all relevant benchmarks for a given research domain and capability focus area.
Input
- research_domain: The broad research area (e.g., "natural language understanding", "code generation", "multimodal reasoning")
- capability_focus: Specific capability of interest (e.g., "commonsense reasoning", "mathematical problem solving")
Procedure
- Search Papers With Code for benchmarks tagged with the domain
- Search Semantic Scholar for benchmark papers in the domain
- Search web for leaderboards and evaluation suites
- For each benchmark found, collect: name, year, paper, size, primary metric, current SOTA, status
- Classify by: capability tested, modality, difficulty level, maintenance status
Output
Structured catalog of benchmarks with metadata sufficient for downstream analysis selection.
Available SOPs
Optional, no fixed order; the final leaf is always a sop.
| SOP | When to use |
|---|---|
| spawn-agent | Spawn a customized CC subagent with full MCP tool access. Used by SOPs that declare execution: subagent. |