nickgallick Competitor Monitoring
Weekly monitoring targets
| Competitor | What to track | Where |
|---|---|---|
| SWE-bench | New versions, methodology changes, adoption | GitHub, papers, X |
| HumanEval+ | Updates, new tasks, citations | GitHub, papers |
| LiveCodeBench | New tasks, methodology updates | GitHub, blog |
| Aider benchmark | New features, agent support changes | GitHub |
| CodeClash | Format changes, community growth | Website, X |
| Any new AI benchmarks | Launch announcements | X, HN, Product Hunt, arXiv |
What to do with competitive intel
- Document in weekly competitive brief
- Identify content opportunities: "SWE-bench launched v2 — here's how Bouts compares"
- Identify feature gaps: "Competitor X added [feature] — should we respond?"
- Share with MaksPM and Nick if strategically significant
Competitive brief format (weekly)
COMPETITIVE BRIEF — Week [N]
New developments:
- [Competitor]: [what changed] [significance: high/medium/low]
Content opportunities:
- [idea]
Feature gaps to consider:
- [gap]
No action needed:
- [item] — [reason]
Response stance
- Treat competitor launches as category validation
- Never attack publicly
- Respond with data and methodology comparison when asked
- Celebrate when others make AI evaluation better — rising tide