How do I install the LLM Evaluation skill?

Run npx skillmds@latest add karstenheld3/llm-evaluation in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

What does the LLM Evaluation skill do?

LLM Evaluation Skill It is listed under AI & ML on SkillMD.

Is LLM Evaluation safe to use?

This skill has not completed SkillMD's automated safety review yet. Independent scanners report: SkillSpector: PASS, Skill Scanner: PASS. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with LLM Evaluation?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is LLM Evaluation free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published LLM Evaluation?

karstenheld3 (@karstenheld3) published this skill. Their other Agent Skills are listed on their SkillMD profile.

LLM Evaluation by karstenheld3 — Agent Skill

Name: LLM Evaluation
Author: karstenheld3

File contents

LLM Evaluation Skill

Evaluate LLM performance through structured testing pipelines.

When to Use This Skill

Apply when:

Testing LLM transcription accuracy (images → text)
Comparing multiple LLM outputs for quality
Measuring model performance with LLM-as-judge
Analyzing API costs across models
Finding optimal concurrency limits for API calls
Running batch evaluations with parallel workers

Do NOT apply when:

Making single ad-hoc LLM calls (use API directly)
Testing non-LLM systems
Simple file processing without LLM involvement

Quick Start

Run SETUP.md once to install dependencies
Choose your workflow:
- Single call: call-llm.py for one-off tests
- Batch processing: call-llm-batch.py for multiple files
- Full pipeline: Transcribe → Questions → Answers → Evaluate
- Worker limits: find-workers-limit.py to find max concurrency
- Cost analysis: analyze-costs.py for token usage reports

Available Scripts

Core Scripts:

call-llm.py - Single LLM API call
call-llm-batch.py - Batch processing with parallel workers
find-workers-limit.py - Discover max concurrent workers

Evaluation Pipeline:

generate-questions.py - Create test questions from source
generate-answers.py - Generate answers from processed text
evaluate-answers.py - Score answers with LLM judge
compare-transcription-runs.py - Compare outputs (Levenshtein/semantic)

Analysis:

analyze-costs.py - Calculate token costs
llm-evaluation-selftest.py - Validate all scripts work

Details: See LLM_EVALUATION_SCRIPTS.md for parameters and examples

Key Findings

Judge model recommendation: Use gpt-5-mini for LLM-as-judge tasks (best calibration on similarity tests)

Worker limits (2026-01-26):

OpenAI models: 120+ concurrent workers
Anthropic models: 60+ concurrent workers

Claude model IDs: Use exact release dates (e.g., claude-opus-4-5-20251101 not 20250514). See LLM_EVALUATION_CLAUDE_MODELS.md for verified IDs.

Tested models: 16+ models validated via llm-evaluation-selftest.py. See LLM_EVALUATION_TESTED_MODELS.md for full list.

---name: llm-evaluationdescription: LLM Evaluation Skill---# LLM Evaluation SkillEvaluate LLM performance through structured testing pipelines.## When to Use This Skill**Apply when:**- Testing LLM transcription accuracy (images → text)- Comparing multiple LLM outputs for quality- Measuring model performance with LLM-as-judge- Analyzing API costs across models- Finding optimal concurrency limits for API calls- Running batch evaluations with parallel workers**Do NOT apply when:**- Making single ad-hoc LLM calls (use API directly)- Testing non-LLM systems- Simple file processing without LLM involvement## Quick Start1. Run `SETUP.md` once to install dependencies2. Choose your workflow: - **Single call**: `call-llm.py` for one-off tests - **Batch processing**: `call-llm-batch.py` for multiple files - **Full pipeline**: Transcribe → Questions → Answers → Evaluate - **Worker limits**: `find-workers-limit.py` to find max concurrency - **Cost analysis**: `analyze-costs.py` for token usage reports## Available Scripts**Core Scripts:**- `call-llm.py` - Single LLM API call- `call-llm-batch.py` - Batch processing with parallel workers- `find-workers-limit.py` - Discover max concurrent workers**Evaluation Pipeline:**- `generate-questions.py` - Create test questions from source- `generate-answers.py` - Generate answers from processed text- `evaluate-answers.py` - Score answers with LLM judge- `compare-transcription-runs.py` - Compare outputs (Levenshtein/semantic)**Analysis:**- `analyze-costs.py` - Calculate token costs- `llm-evaluation-selftest.py` - Validate all scripts work**Details:** See `LLM_EVALUATION_SCRIPTS.md` for parameters and examples## Key Findings**Judge model recommendation:** Use `gpt-5-mini` for LLM-as-judge tasks (best calibration on similarity tests)**Worker limits (2026-01-26):**- OpenAI models: 120+ concurrent workers- Anthropic models: 60+ concurrent workers**Claude model IDs:** Use exact release dates (e.g., `claude-opus-4-5-20251101` not `20250514`). See `LLM_EVALUATION_CLAUDE_MODELS.md` for verified IDs.**Tested models:** 16+ models validated via `llm-evaluation-selftest.py`. See `LLM_EVALUATION_TESTED_MODELS.md` for full list.

Frequently asked questions

LLM Evaluation