LLM Rankings Skill
Comprehensive evaluation and ranking system for comparing language models across performance, cost, and technical dimensions.
Core Capabilities
This skill provides four main ranking methodologies:
- Benchmark-Based Rankings - Objective comparisons using MMLU, GSM8K, HumanEval scores
- Task-Specific Rankings - Weighted recommendations for code generation, creative writing, reasoning, etc.
- Cost-Effectiveness Rankings - Performance per dollar analysis
- Real-World Performance - API reliability, documentation quality, ease of integration
Standard Workflows
Simple Comparison Request
When user asks "Which LLM is better for X?":
- Load relevant benchmark data from
references/benchmarks.md
- Filter models matching requirements
- Calculate rankings with appropriate weighting
- Present top 3-5 recommendations with justification
- Include pricing information from
references/pricing.md
Detailed Analysis Request
When user asks for comprehensive comparison:
- Load model specifications from
references/model-details.md
- Generate side-by-side comparison table
- Include benchmark scores across multiple tests
- Calculate cost projections for expected usage
- Provide deployment considerations
Best Model for Task Query
When user describes a specific use case:
- Parse task requirements (performance needs, budget, technical constraints)
- Map to capability dimensions
- Load task-specific rankings from
references/use-cases.md
- Return top 3 models with detailed reasoning
- Include caveats and alternative suggestions
Reference Resources
Load these files as needed to inform recommendations:
- benchmarks.md - Comprehensive benchmark scores (MMLU, GSM8K, HumanEval, MMMU, etc.)
- model-details.md - Technical specifications, context windows, API details, capabilities
- use-cases.md - Task-specific recommendations organised by common use cases
- pricing.md - Current pricing across all providers, cost optimisation strategies
Output Formats
Quick Recommendation
Present concise recommendations with model name, key strength, pricing snapshot, and one-sentence justification.
Comparison Table
Use markdown tables comparing models across relevant dimensions (performance, context window, pricing, best use).
Detailed Analysis
Structure as:
- Executive summary (2-3 sentences)
- Top recommendations (ranked with justification)
- Performance comparison (benchmark scores)
- Cost analysis (usage projections)
- Implementation considerations
- Alternative options
Key Principles
- Evidence-Based - Support all rankings with benchmark data or documented performance
- Context-Aware - Consider user's specific requirements, budget, technical environment
- Transparent - Explain weighting decisions and ranking criteria clearly
- Current Information - Use web_search to verify latest releases, pricing changes, benchmark updates
- Practical Focus - Prioritise real-world usage factors over pure benchmark scores
- Balanced - Present strengths and weaknesses honestly for each model
Important Considerations
- Benchmark Limitations - Benchmarks don't perfectly reflect real-world performance
- Task Specificity - A model's ranking varies significantly by use case
- Pricing Volatility - API pricing changes frequently; verify for important decisions
- Access Availability - Some models have waitlists or geographic restrictions
- Trade-offs - Larger context windows often mean slower processing
Usage Notes
- Always verify current pricing and availability via web search for recent changes
- Consider user's deployment environment (API vs self-hosted)
- Account for additional costs (vision inputs, fine-tuning, enterprise features)
- Recommend testing on user's specific use case before committing
- Highlight when free tiers or trials are available
Model Coverage
Provides comprehensive coverage of:
- Anthropic: Claude Opus 4.1/4, Sonnet 4.5/4, Haiku 4
- OpenAI: GPT-4 Turbo, GPT-4o, GPT-4o-mini, o1-preview, o1-mini
- Google: Gemini 1.5 Pro, Gemini 1.5 Flash
- Meta: Llama 3.1 (405B, 70B, 8B)
- Mistral: Large 2, Small
- DeepSeek: Coder V2
- Other providers as relevant to user queries
1---2name: llm-rankings3description: Comprehensive LLM model evaluation and ranking system. Use when users ask to compare language models, find the best model for a specific task, understand model capabilities, get pricing information, or need help selecting between GPT-4, Claude, Gemini, Llama, or other LLMs. Provides benchmark-based rankings, cost analysis, and use-case-specific recommendations across reasoning, code generation, long context, multimodal, and other capabilities.4---5
6# LLM Rankings Skill
7
8Comprehensive evaluation and ranking system for comparing language models across performance, cost, and technical dimensions.
9
10## Core Capabilities
11
12This skill provides four main ranking methodologies:
13
141. **Benchmark-Based Rankings** - Objective comparisons using MMLU, GSM8K, HumanEval scores
152. **Task-Specific Rankings** - Weighted recommendations for code generation, creative writing, reasoning, etc.
163. **Cost-Effectiveness Rankings** - Performance per dollar analysis
174. **Real-World Performance** - API reliability, documentation quality, ease of integration
18
19## Standard Workflows
20
21### Simple Comparison Request
22
23When user asks "Which LLM is better for X?":
24
251. Load relevant benchmark data from `references/benchmarks.md`
262. Filter models matching requirements
273. Calculate rankings with appropriate weighting
284. Present top 3-5 recommendations with justification
295. Include pricing information from `references/pricing.md`
30
31### Detailed Analysis Request
32
33When user asks for comprehensive comparison:
34
351. Load model specifications from `references/model-details.md`
362. Generate side-by-side comparison table
373. Include benchmark scores across multiple tests
384. Calculate cost projections for expected usage
395. Provide deployment considerations
40
41### Best Model for Task Query
42
43When user describes a specific use case:
44
451. Parse task requirements (performance needs, budget, technical constraints)
462. Map to capability dimensions
473. Load task-specific rankings from `references/use-cases.md`
484. Return top 3 models with detailed reasoning
495. Include caveats and alternative suggestions
50
51## Reference Resources
52
53Load these files as needed to inform recommendations:
54
55- **benchmarks.md** - Comprehensive benchmark scores (MMLU, GSM8K, HumanEval, MMMU, etc.)
56- **model-details.md** - Technical specifications, context windows, API details, capabilities
57- **use-cases.md** - Task-specific recommendations organised by common use cases
58- **pricing.md** - Current pricing across all providers, cost optimisation strategies
59
60## Output Formats
61
62### Quick Recommendation
63Present concise recommendations with model name, key strength, pricing snapshot, and one-sentence justification.
64
65### Comparison Table
66Use markdown tables comparing models across relevant dimensions (performance, context window, pricing, best use).
67
68### Detailed Analysis
69Structure as:
701. Executive summary (2-3 sentences)
712. Top recommendations (ranked with justification)
723. Performance comparison (benchmark scores)
734. Cost analysis (usage projections)
745. Implementation considerations
756. Alternative options
76
77## Key Principles
78
791. **Evidence-Based** - Support all rankings with benchmark data or documented performance
802. **Context-Aware** - Consider user's specific requirements, budget, technical environment
813. **Transparent** - Explain weighting decisions and ranking criteria clearly
824. **Current Information** - Use web_search to verify latest releases, pricing changes, benchmark updates
835. **Practical Focus** - Prioritise real-world usage factors over pure benchmark scores
846. **Balanced** - Present strengths and weaknesses honestly for each model
85
86## Important Considerations
87
88- **Benchmark Limitations** - Benchmarks don't perfectly reflect real-world performance
89- **Task Specificity** - A model's ranking varies significantly by use case
90- **Pricing Volatility** - API pricing changes frequently; verify for important decisions
91- **Access Availability** - Some models have waitlists or geographic restrictions
92- **Trade-offs** - Larger context windows often mean slower processing
93
94## Usage Notes
95
96- Always verify current pricing and availability via web search for recent changes
97- Consider user's deployment environment (API vs self-hosted)
98- Account for additional costs (vision inputs, fine-tuning, enterprise features)
99- Recommend testing on user's specific use case before committing
100- Highlight when free tiers or trials are available
101
102## Model Coverage
103
104Provides comprehensive coverage of:
105- **Anthropic:** Claude Opus 4.1/4, Sonnet 4.5/4, Haiku 4
106- **OpenAI:** GPT-4 Turbo, GPT-4o, GPT-4o-mini, o1-preview, o1-mini
107- **Google:** Gemini 1.5 Pro, Gemini 1.5 Flash
108- **Meta:** Llama 3.1 (405B, 70B, 8B)
109- **Mistral:** Large 2, Small
110- **DeepSeek:** Coder V2
111- **Other providers** as relevant to user queries
majiayu000