How do I install the LLM Evaluation skill?

Run npx skillmds@latest add kumaran-is/llm-evaluation in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is LLM Evaluation safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with LLM Evaluation?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is LLM Evaluation free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published LLM Evaluation?

kumaran-is (@kumaran-is) published this skill. Their other Agent Skills are listed on their SkillMD profile.

LLM Evaluation by kumaran-is — Agent Skill

Name: LLM Evaluation
Author: kumaran-is

LLM Evaluation

Implement comprehensive evaluation strategies for LLM applications — automated metrics (BLEU, ROUGE, BERTScore, RAG metrics), A/B testing with statistical rigor, regression detection, and benchmarking. Use when measuring agent quality, comparing models or prompts, or building eval pipelines for LangGraph or Google ADK agents.

kumaran-is 6914a90 5 files · 60.9 KB Updated Sep 17, 2026

File contents

Iron Law

NO EVALUATION CONCLUSION WITHOUT A BASELINE — comparing model A to model B means nothing without a defined baseline and statistical test. Every eval claim requires: metric name, sample size, statistical significance (p < 0.05), and effect size.

LLM Evaluation Skill — Python 3.14 + LangGraph + Google ADK

When to Use This Skill

Measuring quality of a RAG agent's retrieval or generation output
Comparing two prompt variants or model versions before promoting to production
Detecting metric regressions in CI after code or model changes
Building an automated eval harness for a LangGraph or Google ADK agent
Selecting the right metric family for a given eval task (retrieval vs generation vs classification)
Establishing baseline scores before any optimization work
Setting up inter-rater agreement to validate human evaluation labels

Metric Selection Quick Reference

Eval Task	Primary Metric	Secondary Metric	When NOT to Use
RAG Retrieval	MRR, NDCG@K	Precision@K, Recall@K	When you have no relevance judgments
Text Generation (summarization)	ROUGE-L, BERTScore-F1	BLEU	When output style matters more than content
Classification	F1 (macro)	Precision/Recall	When class imbalance makes accuracy misleading
A/B Prompt Comparison	Cohen's d effect size	p-value alone	When sample size < 30
RAG Generation Quality	BERTScore + Groundedness	Perplexity	For factual domain QA

Framework Decision Tree

What are you evaluating?
├── Retrieval quality (RAG) → reference/evaluation-metrics.md#rag-metrics
├── Text generation quality → reference/evaluation-metrics.md#text-generation
├── Comparing two prompts/models → reference/ab-testing.md
├── Detecting regression vs baseline → reference/ab-testing.md#regression
├── Building full eval harness → reference/evaluation-harness.md
└── Evaluating agent trajectory (tool calls, reasoning steps) → reference/trajectory-evaluation.md

Reference Files

File	Content	When to Use
`reference/evaluation-metrics.md`	BLEU, ROUGE, BERTScore, RAG metrics (MRR, NDCG, Precision@K), classification metrics, custom groundedness	Implementing metrics for any eval task
`reference/ab-testing.md`	A/B test with t-test + Cohen's d, regression detector class, sample size calculator, CI/CD integration	Comparing prompts/models, detecting regressions
`reference/evaluation-harness.md`	Full eval harness for LangGraph agents and ADK agents, benchmark runner, inter-rater agreement	Building end-to-end eval pipeline
`reference/trajectory-evaluation.md`	4-pillar evaluation (Effectiveness 40%, Efficiency 20%, Robustness 20%, Safety 20%), safety zero-tolerance, EvaluationResult structure, batch eval, pass/fail gates	Evaluating LangGraph agent trajectories end-to-end

Post-Code Review

After implementing any eval pipeline, dispatch the agentic-ai-reviewer agent to verify:

Statistical assumptions are met (sample size, normality, independence)
Baseline is committed and versioned (not ephemeral)
Regression thresholds are documented and justified
No silent metric failures (every metric error must surface, not default to 0.0)

---name: llm-evaluationdescription: Implement comprehensive evaluation strategies for LLM applications — automated metrics (BLEU, ROUGE, BERTScore, RAG metrics), A/B testing with statistical rigor, regression detection, and benchmarking. Use when measuring agent quality, comparing models or prompts, or building eval pipelines for LangGraph or Google ADK agents.---## Iron Law**NO EVALUATION CONCLUSION WITHOUT A BASELINE — comparing model A to model B means nothing without a defined baseline and statistical test. Every eval claim requires: metric name, sample size, statistical significance (p < 0.05), and effect size.**# LLM Evaluation Skill — Python 3.14 + LangGraph + Google ADK## When to Use This Skill- Measuring quality of a RAG agent's retrieval or generation output- Comparing two prompt variants or model versions before promoting to production- Detecting metric regressions in CI after code or model changes- Building an automated eval harness for a LangGraph or Google ADK agent- Selecting the right metric family for a given eval task (retrieval vs generation vs classification)- Establishing baseline scores before any optimization work- Setting up inter-rater agreement to validate human evaluation labels## Metric Selection Quick Reference| Eval Task | Primary Metric | Secondary Metric | When NOT to Use ||-----------|---------------|-----------------|-----------------|| RAG Retrieval | MRR, NDCG@K | Precision@K, Recall@K | When you have no relevance judgments || Text Generation (summarization) | ROUGE-L, BERTScore-F1 | BLEU | When output style matters more than content || Classification | F1 (macro) | Precision/Recall | When class imbalance makes accuracy misleading || A/B Prompt Comparison | Cohen's d effect size | p-value alone | When sample size < 30 || RAG Generation Quality | BERTScore + Groundedness | Perplexity | For factual domain QA |## Framework Decision Tree```What are you evaluating?├── Retrieval quality (RAG) → reference/evaluation-metrics.md#rag-metrics├── Text generation quality → reference/evaluation-metrics.md#text-generation├── Comparing two prompts/models → reference/ab-testing.md├── Detecting regression vs baseline → reference/ab-testing.md#regression├── Building full eval harness → reference/evaluation-harness.md└── Evaluating agent trajectory (tool calls, reasoning steps) → reference/trajectory-evaluation.md```## Reference Files| File | Content | When to Use ||------|---------|-------------|| `reference/evaluation-metrics.md` | BLEU, ROUGE, BERTScore, RAG metrics (MRR, NDCG, Precision@K), classification metrics, custom groundedness | Implementing metrics for any eval task || `reference/ab-testing.md` | A/B test with t-test + Cohen's d, regression detector class, sample size calculator, CI/CD integration | Comparing prompts/models, detecting regressions || `reference/evaluation-harness.md` | Full eval harness for LangGraph agents and ADK agents, benchmark runner, inter-rater agreement | Building end-to-end eval pipeline || `reference/trajectory-evaluation.md` | 4-pillar evaluation (Effectiveness 40%, Efficiency 20%, Robustness 20%, Safety 20%), safety zero-tolerance, EvaluationResult structure, batch eval, pass/fail gates | Evaluating LangGraph agent trajectories end-to-end |## Post-Code ReviewAfter implementing any eval pipeline, dispatch the `agentic-ai-reviewer` agent to verify:- Statistical assumptions are met (sample size, normality, independence)- Baseline is committed and versioned (not ephemeral)- Regression thresholds are documented and justified- No silent metric failures (every metric error must surface, not default to 0.0)

Frequently asked questions