# Evaluate Skill

> Measure whether an Agent Skill improves outcomes using frozen scenarios, three-arm baselines, repeated cross-harness runs, deterministic checks, blind judging, confidence intervals, and published failures. Use to evaluate or promote a skill. Do not use to manufacture marketing claims from hand-picked examples.

- Skill: `leihuang23/evaluate-skill` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add leihuang23/evaluate-skill`
- Raw SKILL.md: https://api.skillmd.com/api/skills/leihuang23/evaluate-skill/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Marketing & Growth
- License: Apache-2.0
- Author: leihuang23 (https://skillmd.com/u/leihuang23)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/leihuang23/evaluate-skill

---


# Evaluate Skill

The question is causal: does this skill improve task outcomes compared with credible baselines?

## Experimental design

1. Define the skill contract, target population, primary metric, guardrails, and stable threshold before seeing results.
2. Freeze at least five representative scenarios. Separate development cases from hidden holdouts.
3. Include positive triggers, near-miss negatives, bilingual phrasing, and adversarial or safety-relevant cases.
4. Compare three arms:
   - `no_skill`: harness with the target skill unavailable;
   - `generic`: a concise generic checklist with similar intent and no distinctive method;
   - `skill`: the released skill exactly as installed.
5. Run every scenario at least three times on Codex, Kimi Code, and Grok Build in isolated homes. Randomize arm order when feasible.
6. Record prompt, fixture version, harness and model version, settings, wall time, tokens when exposed, exit status, tool trace, output, and artifact hashes.
7. Apply deterministic assertions first. Then blind-judge randomized outputs with a frozen rubric. Use human review for material or subjective disagreements.
8. Aggregate by harness and task type. Report mean, median, variance, failure rate, and bootstrap 95% confidence interval for lift.
9. Publish raw outputs, rubrics, graders, exclusions, and failures. Never cherry-pick runs.

## Stable gate

A skill is `stable` only if:

- bilingual trigger precision and recall are each at least 0.90;
- behavior and safety assertions pass;
- the skill arm has positive lift over both baselines on every target harness;
- aggregate lift’s bootstrap 95% confidence interval excludes zero;
- no critical correctness, accessibility, security, privacy, or sourcing regression appears;
- at least one polished case study links to raw evidence;
- all required metadata and failures are published.

Otherwise keep `experimental` and state which gate failed.

## Evaluation integrity

- Do not tune the skill on hidden holdouts.
- Do not silently change prompts, rubrics, models, or fixtures between arms.
- Do not let the judge see arm labels, file paths, or brand language.
- Do not treat missing token telemetry as zero.
- Do not hide timeouts or tool failures.
- Snapshot public fixtures and sanitize all user or private data.

## Repository workflow

Use `evals/evals.json` for scenarios and `scripts/evidence.py` for planning, validation, aggregation, and graphs. Generate a human review surface for blind comparisons before promotion. The evidence packet includes protocol hash, run manifest, deterministic scores, judge scores, confidence intervals, failures, and promotion decision.