dixiyao PinchBench Benchmark Skill
PinchBench measures how well LLM models perform as the brain of an OpenClaw agent. Results are collected on a public leaderboard at pinchbench.com.
Prerequisites
- Python 3.10+
- uv package manager
- OpenClaw instance (this agent)
Quick Start
cd <skill_directory>
# Run benchmark with a specific model
uv run benchmark.py --model anthropic/claude-sonnet-4
# Run only automated tasks (faster)
uv run benchmark.py --model anthropic/claude-sonnet-4 --suite automated-only
# Run specific tasks
uv run benchmark.py --model anthropic/claude-sonnet-4 --suite task_calendar,task_stock
# Skip uploading results
uv run benchmark.py --model anthropic/claude-sonnet-4 --no-upload
Available Tasks (23)
| Task | Category | Description |
|---|---|---|
task_sanity |
Basic | Verify agent works |
task_calendar |
Productivity | Calendar event creation |
task_stock |
Research | Stock price lookup |
task_blog |
Writing | Blog post creation |
task_weather |
Coding | Weather script |
task_summary |
Analysis | Document summarization |
task_events |
Research | Conference research |
task_email |
Writing | Email drafting |
task_memory |
Memory | Context retrieval |
task_files |
Files | File structure creation |
task_workflow |
Integration | Multi-step API workflow |
task_clawdhub |
Skills | ClawHub interaction |
task_skill_search |
Skills | Skill discovery |
task_image_gen |
Creative | Image generation |
task_humanizer |
Writing | Text humanization |
task_daily_summary |
Productivity | Daily digest |
task_email_triage |
Inbox triage | |
task_email_search |
Email search | |
task_market_research |
Research | Market analysis |
task_spreadsheet_summary |
Analysis | Spreadsheet analysis |
task_eli5_pdf_summary |
Analysis | PDF simplification |
task_openclaw_comprehension |
Knowledge | OpenClaw docs comprehension |
task_second_brain |
Memory | Knowledge management |
Command Line Options
| Option | Description |
|---|---|
--model |
Model identifier (e.g., anthropic/claude-sonnet-4) |
--suite |
all, automated-only, or comma-separated task IDs |
--output-dir |
Results directory (default: results/) |
--timeout-multiplier |
Scale task timeouts for slower models |
--runs |
Number of runs per task for averaging |
--no-upload |
Skip uploading to leaderboard |
--register |
Request new API token for submissions |
--upload FILE |
Upload previous results JSON |
Token Registration
To submit results to the leaderboard:
# Register for an API token (one-time)
uv run benchmark.py --register
# Run benchmark (auto-uploads with token)
uv run benchmark.py --model anthropic/claude-sonnet-4
Results
Results are saved as JSON in the output directory:
# View task scores
jq '.tasks[] | {task_id, score: .grading.mean}' results/0001_anthropic-claude-sonnet-4.json
# Show failed tasks
jq '.tasks[] | select(.grading.mean < 0.5)' results/*.json
# Calculate overall score
jq '{average: ([.tasks[].grading.mean] | add / length)}' results/*.json
Adding Custom Tasks
Create a markdown file in tasks/ following TASK_TEMPLATE.md. Each task needs:
- YAML frontmatter (id, name, category, grading_type, timeout)
- Prompt section
- Expected behavior
- Grading criteria
- Automated checks (Python grading function)
Leaderboard
View results at pinchbench.com. The leaderboard shows:
- Model rankings by overall score
- Per-task breakdowns
- Historical performance trends