yogsoth-ai Performance Extraction
Purpose
Extract structured performance data from papers, leaderboards, and reproducibility studies. Each data point is a (Task, Dataset, Metric, Score, Conditions) tuple with full provenance. Prioritizes primary sources (original papers) but cross-references against leaderboards and third-party reproductions.
Budget
| Resource | Floor | Target |
|---|---|---|
| Methods covered | 20 | 30 |
| Data points extracted | 100 | 150 |
| Web searches | 25 | 40 |
| Papers read | 15 | 30 |
State Ledger
<HARD-GATE>
| Metric | Current | Target | Status |
|--------|---------|--------|--------|
| Methods covered | 0 | 30 | BLOCKED |
| Data points extracted | 0 | 150 | BLOCKED |
| Web searches used | 0 | 40 | — |
| Papers read | 0 | 30 | — |
| Datasets covered | 0 | 5 | — |
| Metrics tracked | 0 | 3 | — |
</HARD-GATE>
Cannot exit until data_points >= 120 (80% of target).
Available Tactics
- leaderboard-harvesting — Bulk data collection from structured platforms
Available SOPs
- score-extraction — Extract tuples from individual papers
- condition-cataloging — Record conditions alongside scores
Execution Guidance
- For each method in the inventory, locate the original paper
- Use score-extraction SOP on each paper to pull all reported results
- Cross-reference against Papers With Code / benchmark leaderboards
- Use condition-cataloging to record experimental setup for each score
- Flag scores that lack essential condition information
- Track provenance: which table/figure in which paper
- Prefer results from official implementations over third-party
Output Format
{
"data_points": [
{
"method": "string",
"task": "string",
"dataset": "string",
"split": "test|val|dev",
"metric": "string",
"score": 0.0,
"confidence_interval": [0.0, 0.0],
"conditions": {
"hardware": "string",
"training_data_size": "string",
"hyperparams_reported": true,
"seeds_reported": true,
"compute_budget": "string"
},
"provenance": {
"paper_id": "string",
"table_or_figure": "string",
"is_primary_source": true
}
}
],
"coverage_summary": {
"methods_covered": 0,
"datasets_covered": 0,
"metrics_tracked": [],
"missing_data_flags": []
}
}
Available Tactics
Optional, no fixed order; the final leaf is always a sop.
| Tactic | When to use |
|---|---|
| leaderboard-harvesting | Systematically collect performance data from platforms and papers |
Available SOPs
Optional, no fixed order; the final leaf is always a sop.
| SOP | When to use |
|---|---|
| condition-cataloging | Record evaluation conditions (data splits, hyperparams, hardware, seeds) from a paper |
| score-extraction | Extract (Task, Dataset, Metric, Score, Conditions) tuples from a paper |