Evaluate Skill
The question is causal: does this skill improve task outcomes compared with credible baselines?
Experimental design
- Define the skill contract, target population, primary metric, guardrails, and stable threshold before seeing results.
- Freeze at least five representative scenarios. Separate development cases from hidden holdouts.
- Include positive triggers, near-miss negatives, bilingual phrasing, and adversarial or safety-relevant cases.
- Compare three arms:
no_skill: harness with the target skill unavailable;generic: a concise generic checklist with similar intent and no distinctive method;skill: the released skill exactly as installed.
- Run every scenario at least three times on Codex, Kimi Code, and Grok Build in isolated homes. Randomize arm order when feasible.
- Record prompt, fixture version, harness and model version, settings, wall time, tokens when exposed, exit status, tool trace, output, and artifact hashes.
- Apply deterministic assertions first. Then blind-judge randomized outputs with a frozen rubric. Use human review for material or subjective disagreements.
- Aggregate by harness and task type. Report mean, median, variance, failure rate, and bootstrap 95% confidence interval for lift.
- Publish raw outputs, rubrics, graders, exclusions, and failures. Never cherry-pick runs.
Stable gate
A skill is stable only if:
- bilingual trigger precision and recall are each at least 0.90;
- behavior and safety assertions pass;
- the skill arm has positive lift over both baselines on every target harness;
- aggregate lift’s bootstrap 95% confidence interval excludes zero;
- no critical correctness, accessibility, security, privacy, or sourcing regression appears;
- at least one polished case study links to raw evidence;
- all required metadata and failures are published.
Otherwise keep experimental and state which gate failed.
Evaluation integrity
- Do not tune the skill on hidden holdouts.
- Do not silently change prompts, rubrics, models, or fixtures between arms.
- Do not let the judge see arm labels, file paths, or brand language.
- Do not treat missing token telemetry as zero.
- Do not hide timeouts or tool failures.
- Snapshot public fixtures and sanitize all user or private data.
Repository workflow
Use evals/evals.json for scenarios and scripts/evidence.py for planning, validation, aggregation, and graphs. Generate a human review surface for blind comparisons before promotion. The evidence packet includes protocol hash, run manifest, deterministic scores, judge scores, confidence intervals, failures, and promotion decision.
1---2name: evaluate-skill3description: Measure whether an Agent Skill improves outcomes using frozen scenarios, three-arm baselines, repeated cross-harness runs, deterministic checks, blind judging, confidence intervals, and published failures. Use to evaluate or promote a skill. Do not use to manufacture marketing claims from hand-picked examples.4license: Apache-2.05---67# Evaluate Skill89The question is causal: does this skill improve task outcomes compared with credible baselines?1011## Experimental design12131. Define the skill contract, target population, primary metric, guardrails, and stable threshold before seeing results.142. Freeze at least five representative scenarios. Separate development cases from hidden holdouts.153. Include positive triggers, near-miss negatives, bilingual phrasing, and adversarial or safety-relevant cases.164. Compare three arms:17 - `no_skill`: harness with the target skill unavailable;18 - `generic`: a concise generic checklist with similar intent and no distinctive method;19 - `skill`: the released skill exactly as installed.205. Run every scenario at least three times on Codex, Kimi Code, and Grok Build in isolated homes. Randomize arm order when feasible.216. Record prompt, fixture version, harness and model version, settings, wall time, tokens when exposed, exit status, tool trace, output, and artifact hashes.227. Apply deterministic assertions first. Then blind-judge randomized outputs with a frozen rubric. Use human review for material or subjective disagreements.238. Aggregate by harness and task type. Report mean, median, variance, failure rate, and bootstrap 95% confidence interval for lift.249. Publish raw outputs, rubrics, graders, exclusions, and failures. Never cherry-pick runs.2526## Stable gate2728A skill is `stable` only if:2930- bilingual trigger precision and recall are each at least 0.90;31- behavior and safety assertions pass;32- the skill arm has positive lift over both baselines on every target harness;33- aggregate lift’s bootstrap 95% confidence interval excludes zero;34- no critical correctness, accessibility, security, privacy, or sourcing regression appears;35- at least one polished case study links to raw evidence;36- all required metadata and failures are published.3738Otherwise keep `experimental` and state which gate failed.3940## Evaluation integrity4142- Do not tune the skill on hidden holdouts.43- Do not silently change prompts, rubrics, models, or fixtures between arms.44- Do not let the judge see arm labels, file paths, or brand language.45- Do not treat missing token telemetry as zero.46- Do not hide timeouts or tool failures.47- Snapshot public fixtures and sanitize all user or private data.4849## Repository workflow5051Use `evals/evals.json` for scenarios and `scripts/evidence.py` for planning, validation, aggregation, and graphs. Generate a human review surface for blind comparisons before promotion. The evidence packet includes protocol hash, run manifest, deterministic scores, judge scores, confidence intervals, failures, and promotion decision.
leihuang23