SOTA Tracker

SOTA Tracker

The definitive open-source database of State-of-the-Art AI models.

Auto-updated daily from LMArena, Artificial Analysis, and HuggingFace.

Why This Exists

AI models are released weekly. Keeping track is impossible. This project:

1. Curates authoritative data - LMArena Elo rankings, manual curation for video/image/audio models
2. Updates daily via GitHub Actions
3. Exports to JSON/CSV/SQLite - Use in your own projects
4. Provides multiple interfaces - Static files, REST API, or MCP server

Quick Start: Use the Data

Option 1: Download JSON/CSV

# Latest data (updated daily)
curl -O https://raw.githubusercontent.com/romancircus/sota-tracker-mcp/main/data/sota_export.json
curl -O https://raw.githubusercontent.com/romancircus/sota-tracker-mcp/main/data/sota_export.csv

Option 2: Clone and Query Locally

git clone https://github.com/romancircus/sota-tracker-mcp.git
cd sota-tracker-mcp

# Query with sqlite3
sqlite3 data/sota.db "SELECT name, sota_rank FROM models WHERE category='llm_api' ORDER BY sota_rank LIMIT 10"

# List forbidden/outdated models
sqlite3 data/sota.db "SELECT name, reason, replacement FROM forbidden"

Option 3: Use with Claude Code (Recommended)

The recommended approach for Claude Code users is static file embedding (lower token cost than MCP):

# Set up daily auto-update of CLAUDE.md
cp scripts/update_sota_claude_md.py ~/scripts/

# Enable systemd timer (runs at 6 AM daily)
systemctl --user enable --now sota-update.timer

# Or run manually
python ~/scripts/update_sota_claude_md.py --update

This embeds a compact SOTA summary directly in your ~/.claude/CLAUDE.md file.

Option 4: REST API

# Start the API server
uvicorn rest_api:app --host 0.0.0.0 --port 8000

# Query endpoints
curl "http://localhost:8000/api/v1/models?category=llm_api"
curl "http://localhost:8000/api/v1/forbidden"
curl "http://localhost:8000/api/v1/models/FLUX.1-dev/freshness"

Option 5: MCP Server (Optional)

MCP support is available but disabled by default (higher token cost). To enable:

# Edit .mcp.json to add the server config
cat > .mcp.json << 'EOF'
{
  "mcpServers": {
    "sota-tracker": {
      "command": "python",
      "args": ["server.py"]
    }
  }
}
EOF

Data Sources

Categories

REST API Endpoints

Run Your Own Scraper

# Install dependencies
pip install -r requirements.txt
pip install playwright
playwright install chromium

# Run all scrapers
python scrapers/run_all.py --export

# Output:
# data/sota_export.json
# data/sota_export.csv
# data/lmarena_latest.json

GitHub Actions (Auto-Update)

This repo uses GitHub Actions to:

• Daily: Scrape all sources, update database, commit changes
• Weekly: Create a tagged release with JSON/CSV exports

To enable on your fork:

1. Fork this repo
2. Go to Settings → Actions → Enable workflows
3. Data will auto-update daily at 6 AM UTC

File Structure

sota-tracker-mcp/
├── server.py                    # MCP server (optional)
├── rest_api.py                  # REST API server
├── init_db.py                   # Database initialization + seeding
├── requirements.txt             # Dependencies
├── data/
│   ├── sota.db                  # SQLite database
│   ├── sota_export.json         # Full JSON export
│   ├── sota_export.csv          # CSV export
│   └── forbidden.json           # Outdated models list
├── scrapers/
│   ├── lmarena.py               # LMArena scraper (Playwright)
│   ├── artificial_analysis.py   # AA scraper (Playwright)
│   └── run_all.py               # Unified runner
├── fetchers/
│   ├── huggingface.py           # HuggingFace API
│   └── cache_manager.py         # Smart caching
└── .github/workflows/
    └── daily-scrape.yml         # GitHub Actions workflow

Contributing

Found a model that's missing or incorrectly ranked?

1. For manual additions: Edit init_db.py and submit a PR
2. For scraper improvements: Edit files in scrapers/
3. For new data sources: Add a new scraper and update run_all.py

See CONTRIBUTING.md for full developer setup and PR process.

OpenCode / Agents.md Integration

The repo now supports updating agents.md files for OpenCode agents:

# Update your agents.md with latest SOTA data
python update_agents_md.py

# Minimal version (top 1 model per category, lightweight)
python update_agents_md.py --minimal

# Custom categories and limit
python update_agents_md.py --categories llm_local image_gen --limit 3

# Force refresh from sources first
python update_agents_md.py --refresh

Automation

Add to your cron or systemd timer for daily updates:

# ~: crontab -e
@daily python ~/Apps/sota-tracker-mcp/update_agents_md.py

Or systemd:

# ~/.config/systemd/user/sota-update.service
[Unit]
Description=Update SOTA models for agents
After=network.target

[Service]
ExecStart=%h/Apps/sota-tracker-mcp/update_agents_md.py

[Install]
WantedBy=default.target

# ~/.config/systemd/user/sota-update.timer
[Unit]
Description=Daily SOTA data update
AccuracySec=1h

[Install]
WantedBy=timers.target

# Enable
systemctl --user enable --now sota-update.timer

See CONTRIBUTING.md for full setup guide

Data Attribution & Legal

This project aggregates publicly available benchmark data from third-party sources. We do not claim ownership of rankings, Elo scores, or benchmark results.

Data Sources (Used With Permission)

Disclaimer

• All benchmark scores and rankings are the intellectual work of their respective sources
• This project provides aggregation and tooling, not original benchmark data
• Data is scraped once daily to minimize server load
• If you are a data source and wish to be excluded, please open an issue

Fair Use

This project:

• Aggregates factual data (not copyrightable)
• Adds value through tooling (API server, unified format, forbidden list)
• Attributes all sources with links
• Does not compete commercially with sources
• Respects robots.txt permissions

License

MIT - See LICENSE for details.

The code in this repository is MIT licensed. The data belongs to its respective sources (see attribution above).