# LLM Evaluation

> LLM Evaluation Skill

- Skill: `karstenheld3/llm-evaluation` (Agent Skill, multi-file: 26 files)
- Install (CLI): `npx skillmds@latest add karstenheld3/llm-evaluation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/karstenheld3/llm-evaluation/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: karstenheld3 (https://skillmd.com/u/karstenheld3)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/karstenheld3/llm-evaluation

---

# LLM Evaluation Skill

Evaluate LLM performance through structured testing pipelines.

## When to Use This Skill

**Apply when:**
- Testing LLM transcription accuracy (images → text)
- Comparing multiple LLM outputs for quality
- Measuring model performance with LLM-as-judge
- Analyzing API costs across models
- Finding optimal concurrency limits for API calls
- Running batch evaluations with parallel workers

**Do NOT apply when:**
- Making single ad-hoc LLM calls (use API directly)
- Testing non-LLM systems
- Simple file processing without LLM involvement

## Quick Start

1. Run `SETUP.md` once to install dependencies
2. Choose your workflow:
   - **Single call**: `call-llm.py` for one-off tests
   - **Batch processing**: `call-llm-batch.py` for multiple files
   - **Full pipeline**: Transcribe → Questions → Answers → Evaluate
   - **Worker limits**: `find-workers-limit.py` to find max concurrency
   - **Cost analysis**: `analyze-costs.py` for token usage reports

## Available Scripts

**Core Scripts:**
- `call-llm.py` - Single LLM API call
- `call-llm-batch.py` - Batch processing with parallel workers
- `find-workers-limit.py` - Discover max concurrent workers

**Evaluation Pipeline:**
- `generate-questions.py` - Create test questions from source
- `generate-answers.py` - Generate answers from processed text
- `evaluate-answers.py` - Score answers with LLM judge
- `compare-transcription-runs.py` - Compare outputs (Levenshtein/semantic)

**Analysis:**
- `analyze-costs.py` - Calculate token costs
- `llm-evaluation-selftest.py` - Validate all scripts work

**Details:** See `LLM_EVALUATION_SCRIPTS.md` for parameters and examples

## Key Findings

**Judge model recommendation:** Use `gpt-5-mini` for LLM-as-judge tasks (best calibration on similarity tests)

**Worker limits (2026-01-26):**
- OpenAI models: 120+ concurrent workers
- Anthropic models: 60+ concurrent workers

**Claude model IDs:** Use exact release dates (e.g., `claude-opus-4-5-20251101` not `20250514`). See `LLM_EVALUATION_CLAUDE_MODELS.md` for verified IDs.

**Tested models:** 16+ models validated via `llm-evaluation-selftest.py`. See `LLM_EVALUATION_TESTED_MODELS.md` for full list.