1---2name: llm-evaluation-framework3enabled: true4description: |5 Use when performing llm evaluation framework — provides a structured framework6 for evaluating large language models (LLMs) for production use. Covers7 task-specific benchmarking, safety testing, cost analysis, latency8 measurement, prompt engineering evaluation, and comparison across models to9 select the optimal LLM for a given use case.10required_connections:11 - prefix: llm-provider12 label: "LLM API Provider"13config_fields:14 - key: use_case15 label: "Target Use Case"16 required: true17 placeholder: "e.g., customer support chatbot, code generation, document summarization"18 - key: models_to_evaluate19 label: "Models to Evaluate"20 required: true21 placeholder: "e.g., GPT-4, Claude, Gemini, Llama 3, Mistral"22 - key: deployment_type23 label: "Deployment Type"24 required: false25 placeholder: "e.g., API, self-hosted, fine-tuned"26features:27 - DATA28 - AI29 - LLM30 - EVALUATION31metadata:32 source: https://skillmd.com/skills/cloudthinker-ai/llm-evaluation-framework33---3435# LLM Evaluation Framework3637## Phase 1: Evaluation Criteria Definition381. Define use case requirements39 - [ ] Task type (generation, classification, extraction, summarization, Q&A)40 - [ ] Input characteristics (length, language, domain)41 - [ ] Output requirements (format, length, structure)42 - [ ] Latency requirements (real-time vs. batch)43 - [ ] Throughput requirements (requests per minute)44 - [ ] Cost constraints (per-token or per-request budget)45 - [ ] Privacy and data residency requirements46 - [ ] Safety and content policy requirements472. Define evaluation metrics and weights4849### Evaluation Criteria Weights5051| Criterion | Weight (%) | Minimum Threshold | Measurement Method |52|-----------|-----------|-------------------|-------------------|53| Task quality | % | | Human eval + automated metrics |54| Latency | % | < ms | P50, P95, P99 |55| Cost | % | < $/1K tokens | Token-based pricing |56| Safety | % | Pass all checks | Red team + automated |57| Reliability | % | > % uptime | API availability |58| Context window | % | > K tokens | Model spec |5960## Phase 2: Evaluation Dataset Creation611. Build evaluation datasets62 - [ ] Representative inputs from real use case63 - [ ] Edge cases and adversarial inputs64 - [ ] Multi-language inputs (if applicable)65 - [ ] Long-context inputs (if applicable)66 - [ ] Ground truth / reference outputs for automated scoring67 - [ ] Minimum 100+ examples per task category682. Create human evaluation rubrics693. Define automated evaluation metrics (BLEU, ROUGE, exact match, F1)7071### Dataset Summary7273| Category | Examples | Ground Truth | Difficulty | Purpose |74|----------|---------|-------------|-----------|---------|75| Standard inputs | | Yes/No | Normal | Baseline quality |76| Edge cases | | Yes/No | Hard | Robustness |77| Adversarial | | N/A | Hard | Safety |78| Domain-specific | | Yes/No | Variable | Domain fitness |79| Long context | | Yes/No | Variable | Context handling |8081## Phase 3: Quality Evaluation821. Run task-specific quality benchmarks83 - [ ] Generate outputs for all evaluation examples84 - [ ] Score with automated metrics85 - [ ] Conduct human evaluation on sample86 - [ ] Evaluate instruction following accuracy87 - [ ] Test structured output generation (JSON, XML)88 - [ ] Assess factual accuracy and hallucination rate892. Compare across models9091### Quality Comparison9293| Model | Automated Score | Human Score (1-5) | Hallucination Rate | Instruction Following | Overall Quality |94|-------|----------------|------------------|-------------------|---------------------|----------------|95| | | | % | % | /100 |9697## Phase 4: Performance & Cost Evaluation981. Measure performance characteristics99 - [ ] Time to first token (TTFT)100 - [ ] Tokens per second (throughput)101 - [ ] P50, P95, P99 end-to-end latency102 - [ ] Latency under concurrent load103 - [ ] Rate limit behavior104 - [ ] Context window utilization impact on latency1052. Calculate cost per use case106 - [ ] Input token cost107 - [ ] Output token cost108 - [ ] Total cost per request (average)109 - [ ] Monthly projected cost at expected volume110111### Performance & Cost Comparison112113| Model | TTFT (ms) | P50 (ms) | P95 (ms) | Tokens/sec | Input $/1M | Output $/1M | Cost/Request |114|-------|----------|---------|---------|-----------|-----------|------------|-------------|115| | | | | | $ | $ | $ |116117## Phase 5: Safety & Reliability Evaluation1181. Test safety and content policies119 - [ ] Harmful content generation resistance120 - [ ] PII handling and redaction121 - [ ] Bias in outputs across demographics122 - [ ] Jailbreak resistance123 - [ ] Content policy compliance124 - [ ] Confidentiality (does not leak training data)1252. Test reliability126 - [ ] API uptime and error rates127 - [ ] Consistency of outputs (temperature=0 determinism)128 - [ ] Graceful degradation under load129 - [ ] Rate limit recovery behavior130131## Phase 6: Selection & Recommendation1321. Score each model across all criteria1332. Apply weights to calculate overall score1343. Consider operational factors (vendor relationship, support, SLA)1354. Evaluate fine-tuning potential if needed1365. Make recommendation with justification137138### Final Comparison Matrix139140| Model | Quality | Latency | Cost | Safety | Reliability | Weighted Score | Rank |141|-------|---------|---------|------|--------|-------------|---------------|------|142| | /100 | /100 | /100 | /100 | /100 | /100 | |143144## Counter-Rationalizations145146| Shortcut | Counter | Why |147|----------|---------|-----|148| "We can skip some steps for this case" | Adapt the workflow steps, don't skip them | Skipped steps are where incidents and oversights originate |149| "The user seems to already know what to do" | Complete all workflow phases with the user | The workflow catches blind spots that experience alone misses |150| "This is a minor case, full process is overkill" | Scale the process down, don't turn it off | Minor cases become major when unstructured; the process scales, not disappears |151| "I'll fill in the details later" | Complete each section before moving on | Deferred details are forgotten; real-time capture is more accurate |152| "The template output isn't necessary" | Always produce the structured output format | Structured output enables comparison, audit trails, and handoff to other teams |153154## Output Format155- **Evaluation Dataset**: Test cases with ground truth156- **Quality Report**: Per-model quality metrics and human eval results157- **Performance Report**: Latency and throughput benchmarks158- **Cost Analysis**: Per-model cost projections at expected volume159- **Recommendation**: Selected model with justification and tradeoffs160161## Action Items162- [ ] Define evaluation criteria and weights for use case163- [ ] Build evaluation dataset with representative examples164- [ ] Run quality benchmarks across candidate models165- [ ] Measure performance and calculate costs166- [ ] Conduct safety and reliability testing167- [ ] Select model and document decision rationale168- [ ] Set up monitoring for model quality in production