Improve Skill Quality
Turn a failing or unconvincing evaluation into a targeted fix. The single most common mistake
in this repo is rewriting skill prose in response to a verdict whose real cause was the eval,
the fixtures, or the harness. Classify first, then fix.
When to Use
- An evaluation verdict is a regression, underpowered, or "no credible improvement".
- A skill wins in the isolated arm but not in the plugin arm, or is reported "not activated".
/evaluate reports "Evaluation ran but produced no results".
- A skill scores well but costs too much (tokens, turns, wall time, plugin menu budget).
- Deciding whether to strengthen or retire a persistently weak skill.
When Not to Use
- Creating a new skill from scratch — use
create-skill.
- Creating a new
eval.yaml from scratch — use create-skill-test.
- Changing the harness itself (
eng/skill-validator, eng/vally-adapter, evaluation*.yml).
Inputs
| Input |
Required |
Description |
| Verdict evidence |
Yes |
The /evaluate PR comment, or results.json from the run artifacts |
| Losing trial transcripts |
Yes for content fixes |
Baseline vs. skilled output plus the judge's stated reason |
| Stimulus-vote W/T/L and repeated-run W/T/L |
Yes |
Separates cross-task evidence from reliability |
| Activation status per arm |
Yes |
Isolated and plugin activation are different failures |
Workflow
Step 1: Get the evidence before forming a hypothesis
Read InvestigatingResults.md for how to
download artifacts and read results.json. Extract, per failing stimulus:
- authoritative stimulus-vote W/T/L and separate repeated-run W/T/L
- activation status in the isolated and plugin arms, separately
- the judge's verbatim reason on each losing trial
- whether any trial errored, timed out, or produced empty output
Do not change skill content until you can quote a losing trial and the judge's reason for it. For the
other cause classes the evidence is different: harness failures are diagnosed from the job log and
the spec, and power problems from the trial record — neither has a losing trial to quote, and
demanding one is what sends people rewriting prose instead.
Step 2: Classify the failure
Work down this table and stop at the first row that matches. Rows are ordered by how often the
symptom has been misdiagnosed as a skill-content problem — the fixture row is first because a
fixture failure also presents as a setup or reliability failure and gets misfiled as one.
| Symptom |
Real cause class |
Go to |
| A fixture does not build, is untracked by git, breaks for the wrong reason, or contradicts itself |
Fixture |
Step 4 |
No results.json, "produced no results", or the spec never loaded |
Harness / spec-load |
Step 3 |
| Trials errored, timed out, or returned empty output |
Reliability |
Step 3 |
| Trajectories unmatched, a trial errored, or the summary disagrees — verdict reported inconclusive |
Reliability (not power) |
Step 3 |
| Positive record (e.g. 16W/8T/1L), comparison conclusive, verdict still not a pass |
Statistical power |
Step 5 |
| Skilled arm equals baseline arm by construction |
Eval design |
Step 6 |
| Activated and lost on quality, judge names a concrete defect |
Skill content |
Step 7 |
| Activated in isolation, not in plugin |
Activation / routing |
Step 8 |
| Not activated in either arm |
Frontmatter description |
Step 8 |
| Wins but costs far more than baseline |
Scope and cost |
Step 7 |
A verdict is only a measured result when the comparison was conclusive: adapt.mjs requires zero
errored trials, zero unmatched trajectories, and an agreeing summary before it will report a pass or
a regression. Confirm that before reading a record as a power problem.
Step 3: Rule out harness and reliability causes
See references/eval-triage.md for the full catalogue. The recurring ones:
- The repository gate rejects the deprecated top-level
config: alias, and Vally rejects a spec
that declares both config: and defaults:. Replace the alias with one defaults: block.
- An errored trial is not automatically a fixture problem — judge-side auth and
session.idle
failures look identical from the verdict and need harness fixes, not SDK pins.
expect_tools: [bash] on an advisory question forces a restore or build and turns an answer into
a timeout with no quality gain.
- Genuine code-generation stimuli need roughly 360s; a timeout yields empty output, which fails
every grader and hides the real quality signal.
- Unmatched trajectories, an errored trial, or a summary that disagrees make the comparison
inconclusive: the remaining matched trials are biased, so the record is not a measured null
and must not be read as a power or content problem.
- A generic YAML parser is not the production loader. If Vally rejects a skill eval, or the native
SDK lane rejects an agent eval's executable scenario fields, classify it as harness / spec-load
before changing content. The native agent parser ignores golden references, so validate those
separately with
check_eval_quality.py and deterministic golden-workspace replay.
- A pass with one worker or an enlarged local timeout is not normal execution evidence. Reproduce
with the repository's normal concurrency and declared suite budget; a failure there is a
reliability defect.
Step 4: Verify the fixtures before touching the skill
Run python eng/eval-quality/check_eval_quality.py — it blocks 22 defect classes that can
cost a real result here. Then confirm by hand:
- every fixture behaves as its stimulus assumes — a fixture meant to be healthy builds, and one
meant to be broken fails for the exact reason the stimulus is about and no other;
- every referenced fixture is in the git index (
git ls-files), not merely on disk — .gitignore
has silently swallowed committed coverage fixtures;
- a fixture never states the same fact in two places that disagree — a Cobertura report whose
declared
line-rate, summary totals and <line> elements differ is the canonical case — or the
two arms legitimately read different truths.
- every preservation or scope assertion covers the complete in-scope file set, not one
representative file.
- the golden trajectory and patch pass the deterministic graders, and a realistic broken mutation
fails the grader that is meant to protect the behavior.
Step 5: Check whether the eval could ever have passed
The gate has two independent bars, and confusing them is the usual misdiagnosis:
- Distinct stimuli ≥ 5. Below that the verdict is reported
underpowered — never a pass, never a regression.
- The sign test must reach p ≤ 0.05 over the discordant (non-tie) stimulus votes. Ties are not
discarded silently; they hold the discordant count down.
| discordant stimulus votes |
records that pass |
p |
| ≤ 4 |
none, however good the skill |
≥ 0.0625 |
| 5–7 |
zero losses only (5W/0L) |
0.031 |
| 8 |
one loss survivable (7W/1L) |
0.035 |
So at exactly 5 stimuli a single tie is fatal — it leaves 4 discordant. At 6 stimuli
one tie is survivable (5W/1T/0L); at 7, up to two are (5W/2T/0L). A loss is not.
So a positive record with a failing verdict is a power problem, not a content problem. Fix it by
adding discriminating stimuli. Raising runs measures reliability for the same task and cannot
clear the floor.
Step 6: Check whether the two arms differ at all
An eval that compares the skill against itself measures judge noise:
- A dormancy guard (
expect_activation: false) must not also set constraints.reject_skills.
That makes the skilled arm skill-free, so the activation contract cannot observe a hijack.
Schema version 4 retains the identical-arm comparison for diagnostics but excludes it from
preference inference; unexpected isolated activation still blocks a pass.
- A skill with
disable-model-invocation: true is absent from the model-facing skilled arm, so its
direct eval compares two identical arms regardless of whether graders inspect activation or answer
content. Cover it through consumer outcomes instead; for example, filter-syntax is covered by
run-tests and mtp-hot-reload.
- A grader whose
config is missing its required key enforces nothing, so the stimulus has one
fewer assertion than it appears to.
Step 7: Fix skill content against the losing trial
Only now change the skill. Apply the patterns in
references/writing-for-baseline-delta.md; the ones that
most often flip a loss:
- Replace reference prose the model already knows with decisions it would otherwise get wrong.
- Add stop-conditions so a strong skill does not over-apply — but do not over-correct into
answering more narrowly than the baseline did.
- Scale output structure to input size; a dashboard for an 8-test suite loses to a direct answer.
- Require truthful validation reporting; claiming "Build succeeded" after a failed restore is an
automatic loss.
- Verify load-bearing API claims by compiling or probing, not by reading source.
- For cost regressions, gate rare or expensive paths behind
references/ reads and size any
orchestration to the user's scope.
Step 8: Fix activation
Activation failures are frontmatter and routing failures, not body failures. See
references/eval-triage.md. Summary:
| Failure |
Fix |
| Not activated in any arm |
Put the user's own words in description: symptoms, error codes, artifact names, quoted requests |
| A sibling skill wins the prompt |
Claim the exact ambiguous words in description, and add matching exclusions on both siblings |
| Model answers with no skill at all |
Raise the stakes in the description, de-crowd the plugin menu, verify with the plugin arm |
| Boundary excludes real scenarios |
Re-read every "do not use for" clause against every eval prompt and real workflow phase |
| Description at the 1,024-char ceiling |
Cut restated body content, not trigger phrases; check the plugin menu budget too |
Step 9: Re-validate
dotnet run --project eng/skill-validator/src/SkillValidator.csproj -- check --plugin ./plugins/<plugin>
python eng/eval-quality/check_eval_quality.py
./eng/run-skill-evals.sh <plugin> <skill>
Use the production path at normal worker concurrency and with the declared defaults.timeout:
Vally for skill evals, and skill-validator evaluate for agent evals. For an agent eval, separately
run check_eval_quality.py, apply each golden patch to its materialized fixture, and run the
applicable deterministic file, output, and command graders against the golden result. Do not use a
serial-only pass or a larger ad hoc budget as completion evidence. For broad routing or behavior
changes, collect separate GPT-family and Claude-family results. Do not pool model families into
extra stimulus votes.
Then request the official run by submitting a PR review containing /evaluate (Files changed →
Review changes), which binds the run to the reviewed commit. Before declaring a regression on the
result, confirm the skill payload actually changed — reruns on byte-identical content have shifted
7W/2T/2L to 4W/5T/2L.
Validation
Common Pitfalls
| Pitfall |
Solution |
| Rewriting skill prose in response to an underpowered verdict |
Underpowered means too few distinct stimuli; add discriminating stimuli instead |
Using the deprecated top-level config: alias |
Rename it to defaults: and preserve its settings; the repository gate rejects the alias |
Padding runs to clear the stimulus floor |
Repeats measure reliability for one task; add stimuli |
| Treating an errored trial as fixture nondeterminism |
Read the stderr first; judge-side auth failures need harness fixes |
| Fixing a "wrong" answer that the fixture actually made wrong |
Check fixture self-consistency before blaming the response |
| Strengthening a skill nobody uses and nothing passes |
Weak eval signal plus thin telemetry is a valid retirement case |
| Landing a fix without re-running |
Verify the invoked payload contains the fix; judge noise is real |
References
- references/writing-for-baseline-delta.md — content patterns that beat the unskilled model
- references/eval-triage.md — symptom, cause and fix catalogue with PR citations
- eng/eval-quality/README.md — the 22 structural gate checks and why each exists
- eng/vally-adapter/InvestigatingResults.md — downloading artifacts and reading
results.json. This is the current guide; the similarly-named eng/skill-validator/src/docs/InvestigatingResults.md documents the retired skill-validator evaluate schema and does not describe today's results.
1---2name: improve-skill-quality3description: Diagnoses and fixes skills in the dotnet/skills repository that lose to their own baseline, fail to activate, time out, or return "no credible improvement". Use when an evaluation verdict is a regression or underpowered, when a skill regressed after a change, when /evaluate reports no results, or when deciding whether a weak skill should be strengthened or retired. Do not use for scaffolding a brand-new skill (use create-skill) or a brand-new eval (use create-skill-test).4metadata:5 source: https://skillmd.com/skills/dotnet/improve-skill-quality6---78# Improve Skill Quality910Turn a failing or unconvincing evaluation into a targeted fix. The single most common mistake11in this repo is rewriting skill prose in response to a verdict whose real cause was the eval,12the fixtures, or the harness. Classify first, then fix.1314## When to Use1516- An evaluation verdict is a regression, underpowered, or "no credible improvement".17- A skill wins in the isolated arm but not in the plugin arm, or is reported "not activated".18- `/evaluate` reports "Evaluation ran but produced no results".19- A skill scores well but costs too much (tokens, turns, wall time, plugin menu budget).20- Deciding whether to strengthen or retire a persistently weak skill.2122## When Not to Use2324- Creating a new skill from scratch — use `create-skill`.25- Creating a new `eval.yaml` from scratch — use `create-skill-test`.26- Changing the harness itself (`eng/skill-validator`, `eng/vally-adapter`, `evaluation*.yml`).2728## Inputs2930| Input | Required | Description |31|-------|----------|-------------|32| Verdict evidence | Yes | The `/evaluate` PR comment, or `results.json` from the run artifacts |33| Losing trial transcripts | Yes for content fixes | Baseline vs. skilled output plus the judge's stated reason |34| Stimulus-vote W/T/L and repeated-run W/T/L | Yes | Separates cross-task evidence from reliability |35| Activation status per arm | Yes | Isolated and plugin activation are different failures |3637## Workflow3839### Step 1: Get the evidence before forming a hypothesis4041Read [InvestigatingResults.md](../../../eng/vally-adapter/InvestigatingResults.md) for how to42download artifacts and read `results.json`. Extract, per failing stimulus:4344- authoritative stimulus-vote W/T/L and separate repeated-run W/T/L45- activation status in the **isolated** and **plugin** arms, separately46- the judge's verbatim reason on each losing trial47- whether any trial errored, timed out, or produced empty output4849Do not change skill content until you can quote a losing trial and the judge's reason for it. For the50other cause classes the evidence is different: harness failures are diagnosed from the job log and51the spec, and power problems from the trial record — neither has a losing trial to quote, and52demanding one is what sends people rewriting prose instead.5354### Step 2: Classify the failure5556Work down this table and stop at the first row that matches. Rows are ordered by how often the57symptom has been misdiagnosed as a skill-content problem — the fixture row is first because a58fixture failure also presents as a setup or reliability failure and gets misfiled as one.5960| Symptom | Real cause class | Go to |61|---------|------------------|-------|62| A fixture does not build, is untracked by git, breaks for the wrong reason, or contradicts itself | Fixture | Step 4 |63| No `results.json`, "produced no results", or the spec never loaded | Harness / spec-load | Step 3 |64| Trials errored, timed out, or returned empty output | Reliability | Step 3 |65| Trajectories unmatched, a trial errored, or the summary disagrees — verdict reported inconclusive | Reliability (not power) | Step 3 |66| Positive record (e.g. 16W/8T/1L), comparison conclusive, verdict still not a pass | Statistical power | Step 5 |67| Skilled arm equals baseline arm by construction | Eval design | Step 6 |68| Activated and lost on quality, judge names a concrete defect | Skill content | Step 7 |69| Activated in isolation, not in plugin | Activation / routing | Step 8 |70| Not activated in either arm | Frontmatter description | Step 8 |71| Wins but costs far more than baseline | Scope and cost | Step 7 |7273A verdict is only a *measured* result when the comparison was conclusive: `adapt.mjs` requires zero74errored trials, zero unmatched trajectories, and an agreeing summary before it will report a pass or75a regression. Confirm that before reading a record as a power problem.7677### Step 3: Rule out harness and reliability causes7879See [references/eval-triage.md](references/eval-triage.md) for the full catalogue. The recurring ones:8081- The repository gate rejects the deprecated top-level `config:` alias, and Vally rejects a spec82 that declares both `config:` and `defaults:`. Replace the alias with one `defaults:` block.83- An errored trial is not automatically a fixture problem — judge-side auth and `session.idle`84 failures look identical from the verdict and need harness fixes, not SDK pins.85- `expect_tools: [bash]` on an advisory question forces a restore or build and turns an answer into86 a timeout with no quality gain.87- Genuine code-generation stimuli need roughly 360s; a timeout yields empty output, which fails88 every grader and hides the real quality signal.89- Unmatched trajectories, an errored trial, or a summary that disagrees make the comparison90 **inconclusive**: the remaining matched trials are biased, so the record is not a measured null91 and must not be read as a power or content problem.92- A generic YAML parser is not the production loader. If Vally rejects a skill eval, or the native93 SDK lane rejects an agent eval's executable scenario fields, classify it as harness / spec-load94 before changing content. The native agent parser ignores golden references, so validate those95 separately with `check_eval_quality.py` and deterministic golden-workspace replay.96- A pass with one worker or an enlarged local timeout is not normal execution evidence. Reproduce97 with the repository's normal concurrency and declared suite budget; a failure there is a98 reliability defect.99100### Step 4: Verify the fixtures before touching the skill101102Run `python eng/eval-quality/check_eval_quality.py` — it blocks 22 defect classes that can103cost a real result here. Then confirm by hand:104105- every fixture behaves as its stimulus assumes — a fixture meant to be healthy builds, and one106 meant to be broken fails for the exact reason the stimulus is about and no other;107- every referenced fixture is in the git index (`git ls-files`), not merely on disk — `.gitignore`108 has silently swallowed committed coverage fixtures;109- a fixture never states the same fact in two places that disagree — a Cobertura report whose110 declared `line-rate`, summary totals and `<line>` elements differ is the canonical case — or the111 two arms legitimately read different truths.112- every preservation or scope assertion covers the complete in-scope file set, not one113 representative file.114- the golden trajectory and patch pass the deterministic graders, and a realistic broken mutation115 fails the grader that is meant to protect the behavior.116117### Step 5: Check whether the eval could ever have passed118119The gate has two independent bars, and confusing them is the usual misdiagnosis:1201211. **Distinct stimuli ≥ 5.** Below that the verdict is reported122 `underpowered` — never a pass, never a regression.1232. **The sign test must reach p ≤ 0.05 over the *discordant* (non-tie) stimulus votes.** Ties are not124 discarded silently; they hold the discordant count down.125126| discordant stimulus votes | records that pass | p |127|---:|---|---:|128| ≤ 4 | none, however good the skill | ≥ 0.0625 |129| 5–7 | zero losses only (5W/0L) | 0.031 |130| 8 | one loss survivable (7W/1L) | 0.035 |131132So at exactly 5 stimuli a single tie is fatal — it leaves 4 discordant. At 6 stimuli133one tie is survivable (5W/1T/0L); at 7, up to two are (5W/2T/0L). A loss is not.134135So a positive record with a failing verdict is a power problem, not a content problem. Fix it by136adding **discriminating stimuli**. Raising `runs` measures reliability for the same task and cannot137clear the floor.138139### Step 6: Check whether the two arms differ at all140141An eval that compares the skill against itself measures judge noise:142143- A dormancy guard (`expect_activation: false`) must **not** also set `constraints.reject_skills`.144 That makes the skilled arm skill-free, so the activation contract cannot observe a hijack.145 Schema version 4 retains the identical-arm comparison for diagnostics but excludes it from146 preference inference; unexpected isolated activation still blocks a pass.147- A skill with `disable-model-invocation: true` is absent from the model-facing skilled arm, so its148 direct eval compares two identical arms regardless of whether graders inspect activation or answer149 content. Cover it through consumer outcomes instead; for example, `filter-syntax` is covered by150 `run-tests` and `mtp-hot-reload`.151- A grader whose `config` is missing its required key enforces nothing, so the stimulus has one152 fewer assertion than it appears to.153154### Step 7: Fix skill content against the losing trial155156Only now change the skill. Apply the patterns in157[references/writing-for-baseline-delta.md](references/writing-for-baseline-delta.md); the ones that158most often flip a loss:159160- Replace reference prose the model already knows with decisions it would otherwise get wrong.161- Add stop-conditions so a strong skill does not over-apply — but do not over-correct into162 answering more narrowly than the baseline did.163- Scale output structure to input size; a dashboard for an 8-test suite loses to a direct answer.164- Require truthful validation reporting; claiming "Build succeeded" after a failed restore is an165 automatic loss.166- Verify load-bearing API claims by compiling or probing, not by reading source.167- For cost regressions, gate rare or expensive paths behind `references/` reads and size any168 orchestration to the user's scope.169170### Step 8: Fix activation171172Activation failures are frontmatter and routing failures, not body failures. See173[references/eval-triage.md](references/eval-triage.md). Summary:174175| Failure | Fix |176|---------|-----|177| Not activated in any arm | Put the user's own words in `description`: symptoms, error codes, artifact names, quoted requests |178| A sibling skill wins the prompt | Claim the exact ambiguous words in `description`, and add matching exclusions on **both** siblings |179| Model answers with no skill at all | Raise the stakes in the description, de-crowd the plugin menu, verify with the plugin arm |180| Boundary excludes real scenarios | Re-read every "do not use for" clause against every eval prompt and real workflow phase |181| Description at the 1,024-char ceiling | Cut restated body content, not trigger phrases; check the plugin menu budget too |182183### Step 9: Re-validate184185```bash186dotnet run --project eng/skill-validator/src/SkillValidator.csproj -- check --plugin ./plugins/<plugin>187python eng/eval-quality/check_eval_quality.py188./eng/run-skill-evals.sh <plugin> <skill>189```190191Use the production path at normal worker concurrency and with the declared `defaults.timeout`:192Vally for skill evals, and `skill-validator evaluate` for agent evals. For an agent eval, separately193run `check_eval_quality.py`, apply each golden patch to its materialized fixture, and run the194applicable deterministic file, output, and command graders against the golden result. Do not use a195serial-only pass or a larger ad hoc budget as completion evidence. For broad routing or behavior196changes, collect separate GPT-family and Claude-family results. Do not pool model families into197extra stimulus votes.198199Then request the official run by submitting a PR review containing `/evaluate` (Files changed →200Review changes), which binds the run to the reviewed commit. Before declaring a regression on the201result, confirm the skill payload actually changed — reruns on byte-identical content have shifted2027W/2T/2L to 4W/5T/2L.203204## Validation205206- [ ] For a content fix, a losing trial and the judge's stated reason are quoted in the PR description.207- [ ] The failure was classified before any content was edited.208- [ ] `check_eval_quality.py` and `skill-validator check` both pass.209- [ ] The production skill or agent runner accepts the executable spec.210- [ ] Golden references pass the standalone checker and deterministic replay.211- [ ] The eval completes under normal concurrency and its declared time budget.212- [ ] Golden acceptance and mutation rejection have been demonstrated.213- [ ] Distinct-stimulus count clears the power bar for the target effect and observed tie rate.214- [ ] Isolated **and** plugin activation are both reported.215- [ ] Broad changes have separate GPT-family and Claude-family evidence.216- [ ] The PR body records root cause, fix, and validation so the lesson is reusable.217218## Common Pitfalls219220| Pitfall | Solution |221|---------|----------|222| Rewriting skill prose in response to an underpowered verdict | Underpowered means too few distinct stimuli; add discriminating stimuli instead |223| Using the deprecated top-level `config:` alias | Rename it to `defaults:` and preserve its settings; the repository gate rejects the alias |224| Padding `runs` to clear the stimulus floor | Repeats measure reliability for one task; add stimuli |225| Treating an errored trial as fixture nondeterminism | Read the stderr first; judge-side auth failures need harness fixes |226| Fixing a "wrong" answer that the fixture actually made wrong | Check fixture self-consistency before blaming the response |227| Strengthening a skill nobody uses and nothing passes | Weak eval signal plus thin telemetry is a valid retirement case |228| Landing a fix without re-running | Verify the invoked payload contains the fix; judge noise is real |229230## References231232- [references/writing-for-baseline-delta.md](references/writing-for-baseline-delta.md) — content patterns that beat the unskilled model233- [references/eval-triage.md](references/eval-triage.md) — symptom, cause and fix catalogue with PR citations234- [eng/eval-quality/README.md](../../../eng/eval-quality/README.md) — the 22 structural gate checks and why each exists235- [eng/vally-adapter/InvestigatingResults.md](../../../eng/vally-adapter/InvestigatingResults.md) — downloading artifacts and reading `results.json`. This is the current guide; the similarly-named `eng/skill-validator/src/docs/InvestigatingResults.md` documents the retired `skill-validator evaluate` schema and does not describe today's results.