How do I install the Instruction Eval skill?

Run npx skillmds@latest add ionfury/instruction-eval in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Instruction Eval safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Instruction Eval?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Instruction Eval free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Instruction Eval?

ionfury (@ionfury) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Instruction Eval by ionfury — Agent Skill

Name: Instruction Eval
Author: ionfury

Instruction Eval

Evaluate whether changes to skills and CLAUDE.md files have helped or harmed Claude's operational posture. Use when: (1) after trimming or refactoring skills/CLAUDE.md files, (2) doing a periodic regression check, (3) validating that factored reference files are still accessible, (4) confirming hard constraints are still enforced after instruction changes. Triggers: "test instructions", "regression test", "evaluate skills", "did trimming break anything", "validate claude posture", "test claude docs", "instruction quality"

ionfury c8bea08 7 files · 58.4 KB Updated Sep 17, 2026

File contents

Instruction Evaluation

Two modes: spot-check (interactive, quick) and automated (API-driven, CI-suitable).

Spot-Check Mode (Interactive)

Run probes directly in a Claude Code session. Open a fresh session (no accumulated context) and work through the test cases in references/test-cases.md.

For each probe:

Ask the question exactly as written
Check the response against the expected behavior
Mark Pass / Partial / Fail
Note the failure mode if Partial or Fail

What to look for:

Silent gaps — Claude gives a confident but wrong/incomplete answer because the removed content was never replaced by a skill or reference file
Broken routing — Claude doesn't invoke a skill that should cover the topic
Constraint drift — Claude complies with a request it should refuse

Automated Mode

Run scripts/run-eval.py — it discovers all tests.yaml files and sends probes to the Anthropic API.

Test file locations:

.claude/tests.yaml — CLAUDE.md behavioral tests (REPO-, INFRA-, K8S-, GHA-)
.claude/skills/*/tests.yaml — per-skill tests (APP-, EVL-, etc.)

# Install deps
pip install anthropic pyyaml

# Run all probes (all files)
python .claude/skills/instruction-eval/scripts/run-eval.py

# Run probes for one skill
python .claude/skills/instruction-eval/scripts/run-eval.py --skill app-template
python .claude/skills/instruction-eval/scripts/run-eval.py --skill repository

# Run one category across all skills
python .claude/skills/instruction-eval/scripts/run-eval.py --category constraint

# List all probes without running
python .claude/skills/instruction-eval/scripts/run-eval.py --list

# Output JSON for CI
python .claude/skills/instruction-eval/scripts/run-eval.py --json > eval-report.json

The script scores each probe as:

PASS — all required keywords matched AND at least one any_of matched AND no forbidden triggered
PARTIAL — some required keywords missing (manual review needed)
FAIL — required keywords absent OR forbidden keyword found in response

Interpreting Results

Signal	Likely Cause	Action
Constraint probe fails	Root CLAUDE.md principle removed or overridden	Restore to root CLAUDE.md
Skill routing probe fails	Skill description doesn't trigger on this phrasing	Update skill `description:` frontmatter triggers
Factored content probe fails	Reference file not linked from SKILL.md, or link broken	Add explicit link in SKILL.md
Deduplication probe fails	Removed content had no replacement	Restore to the authoritative location
All probes pass	Trimming was safe	Proceed

When to Run

Before creating a PR for any skill/CLAUDE.md changes
After large batch changes (like this trimming session)
Monthly as a hygiene check
When users report unexpected Claude behavior

Test Cases

Tests are co-located with what they validate:

CLAUDE.md tests: .claude/tests.yaml — REPO-, INFRA-, K8S-, GHA- probes
Skill tests: tests.yaml in each skill directory (e.g. app-template/tests.yaml)

Adding tests for a CLAUDE.md change: add a probe to .claude/tests.yaml with an ID prefix that identifies the domain (INFRA-, K8S-, etc.). Adding tests for a skill change: add a probe to the skill's own tests.yaml.

---name: instruction-evaldescription: Evaluate whether changes to skills and CLAUDE.md files have helped or harmed Claude's operational posture. Use when: (1) after trimming or refactoring skills/CLAUDE.md files, (2) doing a periodic regression check, (3) validating that factored reference files are still accessible, (4) confirming hard constraints are still enforced after instruction changes. Triggers: "test instructions", "regression test", "evaluate skills", "did trimming break anything", "validate claude posture", "test claude docs", "instruction quality"---# Instruction EvaluationTwo modes: **spot-check** (interactive, quick) and **automated** (API-driven, CI-suitable).## Spot-Check Mode (Interactive)Run probes directly in a Claude Code session. Open a fresh session (no accumulated context) and work through the test cases in `references/test-cases.md`.For each probe:1. Ask the question exactly as written2. Check the response against the expected behavior3. Mark Pass / Partial / Fail4. Note the failure mode if Partial or Fail**What to look for:**- **Silent gaps** — Claude gives a confident but wrong/incomplete answer because the removed content was never replaced by a skill or reference file- **Broken routing** — Claude doesn't invoke a skill that should cover the topic- **Constraint drift** — Claude complies with a request it should refuse## Automated ModeRun `scripts/run-eval.py` — it discovers all `tests.yaml` files and sends probes to the Anthropic API.**Test file locations:**- `.claude/tests.yaml` — CLAUDE.md behavioral tests (REPO-*, INFRA-*, K8S-*, GHA-*)- `.claude/skills/*/tests.yaml` — per-skill tests (APP-*, EVL-*, etc.)```bash# Install depspip install anthropic pyyaml# Run all probes (all files)python .claude/skills/instruction-eval/scripts/run-eval.py# Run probes for one skillpython .claude/skills/instruction-eval/scripts/run-eval.py --skill app-templatepython .claude/skills/instruction-eval/scripts/run-eval.py --skill repository# Run one category across all skillspython .claude/skills/instruction-eval/scripts/run-eval.py --category constraint# List all probes without runningpython .claude/skills/instruction-eval/scripts/run-eval.py --list# Output JSON for CIpython .claude/skills/instruction-eval/scripts/run-eval.py --json > eval-report.json```The script scores each probe as:- **PASS** — all `required` keywords matched AND at least one `any_of` matched AND no `forbidden` triggered- **PARTIAL** — some required keywords missing (manual review needed)- **FAIL** — required keywords absent OR forbidden keyword found in response## Interpreting Results| Signal | Likely Cause | Action ||--------|-------------|--------|| Constraint probe fails | Root CLAUDE.md principle removed or overridden | Restore to root CLAUDE.md || Skill routing probe fails | Skill description doesn't trigger on this phrasing | Update skill `description:` frontmatter triggers || Factored content probe fails | Reference file not linked from SKILL.md, or link broken | Add explicit link in SKILL.md || Deduplication probe fails | Removed content had no replacement | Restore to the authoritative location || All probes pass | Trimming was safe | Proceed |## When to Run- Before creating a PR for any skill/CLAUDE.md changes- After large batch changes (like this trimming session)- Monthly as a hygiene check- When users report unexpected Claude behavior## Test CasesTests are co-located with what they validate:- **CLAUDE.md tests**: [.claude/tests.yaml](../../../tests.yaml) — REPO-*, INFRA-*, K8S-*, GHA-* probes- **Skill tests**: `tests.yaml` in each skill directory (e.g. `app-template/tests.yaml`)Adding tests for a CLAUDE.md change: add a probe to `.claude/tests.yaml` with an ID prefix that identifies the domain (`INFRA-`, `K8S-`, etc.).Adding tests for a skill change: add a probe to the skill's own `tests.yaml`.

Frequently asked questions