SEP-2640 Skill Evaluation (cross-cutting/sep2640/skill-evaluation)
Use when the task is evaluating a delivered skill against SEP-2640-style
conformance and quality criteria: conformance checks, behavioral
coverage, quality scoring, and an acceptance verdict.
Domain quick reference
- Evaluation mirrors the SEP-2640 delivery shape (skills over MCP,
agentskills.io content form); the spec is emerging, so evaluation
targets the stable conformance surface: frontmatter fields,
description with trigger, license, and standards references.
- Conformance checks are boolean gates; one failure fails the package.
- Quality criteria are scored 0.0-1.0 each and combined by weighted
mean; weights must sum to 1.0.
- Coverage ratio = tested behaviors / total behaviors, in 0.0-1.0.
- Acceptance verdict by weighted score and threshold:
score >= accept threshold -> accept; score >= rework threshold
(but below accept) -> rework; below both -> reject.
- Verdicts stay deterministic: same inputs, same outputs, offline.
Workflow
- Run the conformance checks on the delivered package (frontmatter
present, name kebab-case matching the folder, description with a
trigger clause, license, standards references).
- Combine the pass/fail checks into a conformance verdict dict and
decide whether the package is conformant (all pass) or not.
- Inspect the contract test: does it exercise the core logic of the
skill? Score behavioral coverage (tested versus total behavior).
- Score the deterministic quality criteria (offline, no network,
stdlib only) and compute the weighted total.
- Issue the acceptance verdict from the weighted score and the
thresholds, then report the coverage ratio.
Pitfalls
- Failing one conformance check but declaring the package conformant.
- Weights that do not sum to 1.0 silently skewing the total.
- Scoring a contract test that imports non-stdlib modules as
deterministic and offline.
- Accepting a skill whose contract test never calls the core logic.
- Reject threshold above the accept threshold (accept then becomes
unreachable).
Behavior contract (gate 3)
The conformance, scoring, coverage, and verdict logic is exercised by
the gate 3 contract test: scripts/test_evaluation.py against
scripts/evaluation_logic.py (stdlib unittest, offline). Run:
python3 scripts/test_evaluation.py
Compliance
- SEP-2640 is an open specification; quote with citation and note the
status (emerging, not yet stable) per standards-map.yaml.
- compliance: STANDARDS-REF, gated: false.
1---2name: skill-evaluation3description: Use when you must evaluate a delivered skill against SEP-2640-style conformance and quality criteria: run conformance checks on the package (frontmatter, description with trigger, license, standards references), judge whether the skill's contract test exercises the core logic, score deterministic quality criteria (offline, no network, stdlib only), compute coverage ratio of tested versus total behavior, and issue an acceptance verdict (accept, rework, or reject). SEP-2640 stays an emerging spec and an adapter over the agentskills.io content format, so evaluation reuses the same conformance shape as skill delivery. Trigger: skill evaluation, SEP-2640, conformance check, acceptance verdict, quality score, coverage ratio, skill review.4license: Apache-2.05---67# SEP-2640 Skill Evaluation (cross-cutting/sep2640/skill-evaluation)89Use when the task is evaluating a delivered skill against SEP-2640-style10conformance and quality criteria: conformance checks, behavioral11coverage, quality scoring, and an acceptance verdict.1213## Domain quick reference1415- Evaluation mirrors the SEP-2640 delivery shape (skills over MCP,16 agentskills.io content form); the spec is emerging, so evaluation17 targets the stable conformance surface: frontmatter fields,18 description with trigger, license, and standards references.19- Conformance checks are boolean gates; one failure fails the package.20- Quality criteria are scored 0.0-1.0 each and combined by weighted21 mean; weights must sum to 1.0.22- Coverage ratio = tested behaviors / total behaviors, in 0.0-1.0.23- Acceptance verdict by weighted score and threshold:24 score >= accept threshold -> accept; score >= rework threshold25 (but below accept) -> rework; below both -> reject.26- Verdicts stay deterministic: same inputs, same outputs, offline.2728## Workflow29301. Run the conformance checks on the delivered package (frontmatter31 present, name kebab-case matching the folder, description with a32 trigger clause, license, standards references).332. Combine the pass/fail checks into a conformance verdict dict and34 decide whether the package is conformant (all pass) or not.353. Inspect the contract test: does it exercise the core logic of the36 skill? Score behavioral coverage (tested versus total behavior).374. Score the deterministic quality criteria (offline, no network,38 stdlib only) and compute the weighted total.395. Issue the acceptance verdict from the weighted score and the40 thresholds, then report the coverage ratio.4142## Pitfalls4344- Failing one conformance check but declaring the package conformant.45- Weights that do not sum to 1.0 silently skewing the total.46- Scoring a contract test that imports non-stdlib modules as47 deterministic and offline.48- Accepting a skill whose contract test never calls the core logic.49- Reject threshold above the accept threshold (accept then becomes50 unreachable).5152## Behavior contract (gate 3)5354The conformance, scoring, coverage, and verdict logic is exercised by55the gate 3 contract test: scripts/test_evaluation.py against56scripts/evaluation_logic.py (stdlib unittest, offline). Run:57python3 scripts/test_evaluation.py5859## Compliance6061- SEP-2640 is an open specification; quote with citation and note the62 status (emerging, not yet stable) per standards-map.yaml.63- compliance: STANDARDS-REF, gated: false.
ashfordeOU