Lintel on GitHub Copilot. Generated from skills/eval/SKILL.md; edit the canonical file, then run
li-copilot init.
- Resource root:
../../.. from this skill's base directory (the Lintel source with bin/,
lib/, skills/). Write plans, state and evidence into the working repository's .claude/
tree, never into the resource root.
- Skill-relative paths: this skill's own
scripts/, references/ and data/ folders (and a
<base> that the workflow defines as its own directory) mean
../../../skills/eval/ in the Lintel source, not this generated folder.
${LINTEL_SKILLS_DIR:-skills} means the skills root, ../../../skills. A bin/li-run step
runs in the working repository, so use $LINTEL_SKILLS_DIR/eval/ there.
- Shell steps: run Bash snippets with Bash (Git for Windows'
bash.exe on Windows, never
System32\bash.exe). Save a snippet to a temporary .sh file and run
bash "<resource root>/bin/li-run" <file>; it prepares LINTEL_SOURCE_ROOT, LINTEL_REPO_ROOT
and the profile context.
- Tools: Read=
view, Write=create, Edit=edit, Bash=bash/powershell, Grep=grep,
Glob=glob, AskUserQuestion=ask_user, TodoWrite=the plan checklist, Task or a named role=task
with that custom agent, WebFetch=web_fetch.
- Other Lintel workflows are native skills: invoke
/li-<name> rather than reading their
files. Named roles such as CodeReviewer are custom agents.
/li-eval
Apply a selected voice rubric to known-good and known-bad corpus paragraphs, compare
observed verdicts to each verdict_label, and report separate per-cell accuracy.
Preserve the useful calibration method without claiming an automatic evaluation service.
This is a workflow, not proof that a model runner, gateway, audit writer or ship gate
exists. Use actual permitted evaluation or explicitly attributed manual/recorded verdicts.
No corpus is configured in the neutral pack. Applicable project/pack policy may require
evaluation; this skill neither invents that requirement nor clears policy or review gates.
When to use
- Voice corpus is populated, ready to calibrate
- Corpus, rubric, evaluator or settings changed — re-evaluate the affected scope
- The applicable project/pack policy requests calibration evidence
- TEST rubric was tuned — re-eval to measure if accuracy improved
- After cell-coverage expansion (added more paragraphs to a cell)
When NOT to use
- Corpus is empty or absent — report insufficient input, not a passing empty run
- Rubric or permitted evaluator is missing — retain NOT_EVALUATED, not fabricated verdicts
- Single-paragraph spot-check — apply the rubric directly
Inputs
--corpus <path> — explicitly selected corpus, or its path from the existing verified
profile's voice.corpus. Read only authorized inputs; do not scan private packs.
--test <path> — explicit rubric and its version/content identity; no guessed sibling filename
--cells <list> — optional exact cell IDs, not globs or code; default all declared cells
--out <path> — explicitly owned target report such as reports/voice/CALIBRATION.md;
otherwise report in the response only. Never default to writing beside a private corpus.
--threshold <pct> — per-class accuracy threshold, default 90 as an advisory comparison
when policy has not specified one. Reject nonnumeric/nonfinite values or values outside
0..100. Do not relax a policy requirement to make the report pass.
- Evaluation evidence — actual authorized tool/actor and settings, or explicitly supplied
recorded/manual verdicts bound to these inputs. No undeclared
--cli dispatcher is implied.
Workflow
- Bind the scope. Record corpus/rubric paths and content identities, declared cells,
selected IDs, threshold and actual evaluator/settings. Keep those fixed for a run.
Reject unknown/repeated cell IDs, duplicate paragraph IDs, empty/malformed inputs and
labels other than the declared known-good/known-bad ground truth. Structured YAML/JSON
can use the existing
lib/envelope_contract.py load_text reader through a permitted
Python operation from the trusted source. Report parser/dependency errors; do not
build a second corpus parser or use this reader to reinterpret profile policy.
- Apply the actual rubric. For each selected paragraph, obtain an observed verdict
through an available permitted operation, or document the manual/recorded evidence.
Keep expected labels separate from evaluation input where possible; copying ground
truth into the observed column is not evaluation. Preserve the rubric's real detail
(for example technique, kind/daring/deep scores and rule violations) without inventing
fields the rubric does not define.
- Compare per paragraph. Retain paragraph ID, expected label, observed verdict,
evidence reference and discrepancy. Ambiguous, missing or malformed verdicts are
NOT_EVALUATED until resolved, not silent correct answers or discarded rows.
- Compare per cell. Known-good accuracy is correct known-good verdicts divided by
all expected known-good paragraphs; known-bad accuracy uses its own denominator.
Show both fractions, evaluated/expected counts and the threshold. Never merge the
classes so abundant good examples can hide failure on bad examples.
A cell with fewer than two paragraphs of either class is INSUFFICIENT; do not divide
by zero. Any unevaluated paragraph prevents PASS. With sufficient complete evidence,
PASS requires both accuracies to meet the selected threshold; otherwise FAIL.
- Aggregate without promoting scope. List every declared cell as selected/evaluated,
failed, insufficient or not evaluated. Any subset run is PARTIAL_SCOPE, even if every
selected cell passes. An empty corpus has no passing aggregate. ALL_CELLS_PASS applies
only to a nonempty full-corpus run with sufficient complete evidence and every cell
passing; it means comparison success, not release clearance or automatic certification.
- Persist only within authority. Write the requested target report or return it in
conversation. Preserve earlier evidence rather than overwriting a broader report with
a subset. Record actual checks and limits; do not stamp a fictitious audit event or run.
Report format
Include actual scope/input identities, actor/tool or manual evidence, evaluated/expected
counts, both per-class fractions, failures and the exact owned output (or response-only).
Label illustrations as illustrations. Do not name a model/version that was not actually used.
Synthetic worked comparison, not a model run, at threshold 90:
| Cell |
Correct good / expected good |
Correct bad / expected bad |
Status |
| R1 |
2/2 |
2/2 |
PASS |
| R2 |
1/2 |
2/2 |
FAIL |
| R3 |
0/0 |
0/0 |
INSUFFICIENT |
All eight populated paragraphs have recorded verdicts in this example. R2 good accuracy
is 50%, not rescued by its 100% bad accuracy. R3 is declared but empty; no division or
vacuous PASS is valid. Aggregate: one passing, one failed, one insufficient cell; full
corpus calibration is not established.
A useful failure entry preserves the ID and concrete error: R2-GOOD-002, expected
known-good, observed known-bad, with the actual rubric rationale/evidence. Revise the
rubric or examples only within scope, then rerun affected work under the new input
identity. Do not remove failed cells merely to obtain a passing report.
Compliance integration
Apply actual P07 profile requirements and P05 control/review rules when they govern the
authorized work. A malformed required profile is not optional/neutral success. A profile
gateway declaration is not a callable tool or permission to transmit corpus data.
Inspect real evaluator tools before invoking them. Missing/denied execution retains the
manual or NOT_EVALUATED route without bypassing denial. No automatic audit logging,
snapshot stamping, independent actor or downstream enforcement is promised by this method.
An actual downstream consumer must validate its own applicable evidence; this report does
not alter ship controls. Do not launch a paid benchmark to demonstrate the example.
Failure modes
- Invalid corpus/rubric/selection: report the offending input, stop that comparison and
preserve a prior good report. Never replace it with empty success.
- Missing corpus or insufficient class coverage: list what is missing; no calibration claim.
- Unavailable evaluator, permission denial or failed call: preserve NOT_EVALUATED;
retries require existing execution/cost authority and must not erase the original failure.
- Partial rerun or changed inputs: report only that evidence. Prior results may be
referenced separately with their exact unchanged inputs/settings, not counted as executed
in this run or silently promoted into a new whole-corpus result.
- Unowned/existing output conflict: return the result without clobbering the artifact;
obtain specific revision authority before replacing it.
Examples
Full eval:
> /li-eval --corpus fixtures/voice.yaml --test fixtures/rubric.md --out reports/voice/CALIBRATION.md
[Only after those inputs, output and an actual evaluation method are authorized]
Compare every declared cell, retain actual verdict evidence, then report real counts.
If no evaluator/recorded evidence is available: NOT_EVALUATED. No calls or readiness invented.
Cell-scoped re-eval:
> /li-eval --corpus fixtures/voice.yaml --test fixtures/rubric-v2.md --cells R2
Illustration: R2 now has 2/2 correct good and 2/2 correct bad.
Selected cell: PASS. Run: PARTIAL_SCOPE (1 of 3 declared cells).
R1 and R3 were not re-evaluated; no whole-corpus certification or ship clearance.
Stricter threshold:
> /li-eval --threshold 95
[With the same explicitly selected inputs and real evidence]
19/20 good and 20/20 bad meet 95%; 18/20 good does not.
Report the actual scope. A mandatory threshold cannot be relaxed by this skill.
See also
- the pack's voice corpus — the input (
resolve_pack_field voice.corpus)
- the pack's voice test rubric — the rubric
- the calibration file — the output
- the applicable verified profile and review/control contract — separate evidence obligations
- ADR-0021 — a staged general eval harness,
not proof that this voice method already has an automated runner
1---2name: li-eval3description: Run the active pack's voice TEST against its voice CORPUS — per-cell accuracy → CALIBRATION.md.4metadata:5 source: https://skillmd.com/skills/jokerman89/li-eval6---78> **Lintel on GitHub Copilot.** Generated from `skills/eval/SKILL.md`; edit the canonical file, then run9> `li-copilot init`.10> - **Resource root:** `../../..` from this skill's base directory (the Lintel source with `bin/`,11> `lib/`, `skills/`). Write plans, state and evidence into the working repository's `.claude/`12> tree, never into the resource root.13> - **Skill-relative paths:** this skill's own `scripts/`, `references/` and `data/` folders (and a14> `<base>` that the workflow defines as its own directory) mean15> `../../../skills/eval/` in the Lintel source, not this generated folder.16> `${LINTEL_SKILLS_DIR:-skills}` means the skills root, `../../../skills`. A `bin/li-run` step17> runs in the working repository, so use `$LINTEL_SKILLS_DIR/eval/` there.18> - **Shell steps:** run Bash snippets with Bash (Git for Windows' `bash.exe` on Windows, never19> `System32\bash.exe`). Save a snippet to a temporary `.sh` file and run20> `bash "<resource root>/bin/li-run" <file>`; it prepares `LINTEL_SOURCE_ROOT`, `LINTEL_REPO_ROOT`21> and the profile context.22> - **Tools:** Read=`view`, Write=`create`, Edit=`edit`, Bash=`bash`/`powershell`, Grep=`grep`,23> Glob=`glob`, AskUserQuestion=`ask_user`, TodoWrite=the plan checklist, Task or a named role=`task`24> with that custom agent, WebFetch=`web_fetch`.25> - **Other Lintel workflows** are native skills: invoke `/li-<name>` rather than reading their26> files. Named roles such as `CodeReviewer` are custom agents.2728# /li-eval2930Apply a selected voice rubric to known-good and known-bad corpus paragraphs, compare31observed verdicts to each `verdict_label`, and report separate per-cell accuracy.32Preserve the useful calibration method without claiming an automatic evaluation service.3334This is a workflow, not proof that a model runner, gateway, audit writer or ship gate35exists. Use actual permitted evaluation or explicitly attributed manual/recorded verdicts.36No corpus is configured in the neutral pack. Applicable project/pack policy may require37evaluation; this skill neither invents that requirement nor clears policy or review gates.3839## When to use4041- Voice corpus is populated, ready to calibrate42- Corpus, rubric, evaluator or settings changed — re-evaluate the affected scope43- The applicable project/pack policy requests calibration evidence44- TEST rubric was tuned — re-eval to measure if accuracy improved45- After cell-coverage expansion (added more paragraphs to a cell)4647## When NOT to use4849- Corpus is empty or absent — report insufficient input, not a passing empty run50- Rubric or permitted evaluator is missing — retain NOT_EVALUATED, not fabricated verdicts51- Single-paragraph spot-check — apply the rubric directly5253## Inputs5455- `--corpus <path>` — explicitly selected corpus, or its path from the existing verified56 profile's `voice.corpus`. Read only authorized inputs; do not scan private packs.57- `--test <path>` — explicit rubric and its version/content identity; no guessed sibling filename58- `--cells <list>` — optional exact cell IDs, not globs or code; default all declared cells59- `--out <path>` — explicitly owned target report such as `reports/voice/CALIBRATION.md`;60 otherwise report in the response only. Never default to writing beside a private corpus.61- `--threshold <pct>` — per-class accuracy threshold, default 90 as an advisory comparison62 when policy has not specified one. Reject nonnumeric/nonfinite values or values outside63 0..100. Do not relax a policy requirement to make the report pass.64- Evaluation evidence — actual authorized tool/actor and settings, or explicitly supplied65 recorded/manual verdicts bound to these inputs. No undeclared `--cli` dispatcher is implied.6667## Workflow68691. **Bind the scope.** Record corpus/rubric paths and content identities, declared cells,70 selected IDs, threshold and actual evaluator/settings. Keep those fixed for a run.71 Reject unknown/repeated cell IDs, duplicate paragraph IDs, empty/malformed inputs and72 labels other than the declared known-good/known-bad ground truth. Structured YAML/JSON73 can use the existing `lib/envelope_contract.py` `load_text` reader through a permitted74 Python operation from the trusted source. Report parser/dependency errors; do not75 build a second corpus parser or use this reader to reinterpret profile policy.762. **Apply the actual rubric.** For each selected paragraph, obtain an observed verdict77 through an available permitted operation, or document the manual/recorded evidence.78 Keep expected labels separate from evaluation input where possible; copying ground79 truth into the observed column is not evaluation. Preserve the rubric's real detail80 (for example technique, kind/daring/deep scores and rule violations) without inventing81 fields the rubric does not define.823. **Compare per paragraph.** Retain paragraph ID, expected label, observed verdict,83 evidence reference and discrepancy. Ambiguous, missing or malformed verdicts are84 NOT_EVALUATED until resolved, not silent correct answers or discarded rows.854. **Compare per cell.** Known-good accuracy is correct known-good verdicts divided by86 all expected known-good paragraphs; known-bad accuracy uses its own denominator.87 Show both fractions, evaluated/expected counts and the threshold. Never merge the88 classes so abundant good examples can hide failure on bad examples.89 A cell with fewer than two paragraphs of either class is INSUFFICIENT; do not divide90 by zero. Any unevaluated paragraph prevents PASS. With sufficient complete evidence,91 PASS requires both accuracies to meet the selected threshold; otherwise FAIL.925. **Aggregate without promoting scope.** List every declared cell as selected/evaluated,93 failed, insufficient or not evaluated. Any subset run is PARTIAL_SCOPE, even if every94 selected cell passes. An empty corpus has no passing aggregate. ALL_CELLS_PASS applies95 only to a nonempty full-corpus run with sufficient complete evidence and every cell96 passing; it means comparison success, not release clearance or automatic certification.976. **Persist only within authority.** Write the requested target report or return it in98 conversation. Preserve earlier evidence rather than overwriting a broader report with99 a subset. Record actual checks and limits; do not stamp a fictitious audit event or run.100101## Report format102103Include actual scope/input identities, actor/tool or manual evidence, evaluated/expected104counts, both per-class fractions, failures and the exact owned output (or response-only).105Label illustrations as illustrations. Do not name a model/version that was not actually used.106107Synthetic worked comparison, **not a model run**, at threshold 90:108109| Cell | Correct good / expected good | Correct bad / expected bad | Status |110|---|---|---|---|111| R1 | 2/2 | 2/2 | PASS |112| R2 | 1/2 | 2/2 | FAIL |113| R3 | 0/0 | 0/0 | INSUFFICIENT |114115All eight populated paragraphs have recorded verdicts in this example. R2 good accuracy116is 50%, not rescued by its 100% bad accuracy. R3 is declared but empty; no division or117vacuous PASS is valid. Aggregate: one passing, one failed, one insufficient cell; full118corpus calibration is **not established**.119120A useful failure entry preserves the ID and concrete error: `R2-GOOD-002`, expected121known-good, observed known-bad, with the actual rubric rationale/evidence. Revise the122rubric or examples only within scope, then rerun affected work under the new input123identity. Do not remove failed cells merely to obtain a passing report.124125## Compliance integration126127Apply actual P07 profile requirements and P05 control/review rules when they govern the128authorized work. A malformed required profile is not optional/neutral success. A profile129gateway declaration is not a callable tool or permission to transmit corpus data.130131Inspect real evaluator tools before invoking them. Missing/denied execution retains the132manual or NOT_EVALUATED route without bypassing denial. No automatic audit logging,133snapshot stamping, independent actor or downstream enforcement is promised by this method.134An actual downstream consumer must validate its own applicable evidence; this report does135not alter ship controls. Do not launch a paid benchmark to demonstrate the example.136137## Failure modes138139- **Invalid corpus/rubric/selection:** report the offending input, stop that comparison and140 preserve a prior good report. Never replace it with empty success.141- **Missing corpus or insufficient class coverage:** list what is missing; no calibration claim.142- **Unavailable evaluator, permission denial or failed call:** preserve NOT_EVALUATED;143 retries require existing execution/cost authority and must not erase the original failure.144- **Partial rerun or changed inputs:** report only that evidence. Prior results may be145 referenced separately with their exact unchanged inputs/settings, not counted as executed146 in this run or silently promoted into a new whole-corpus result.147- **Unowned/existing output conflict:** return the result without clobbering the artifact;148 obtain specific revision authority before replacing it.149150## Examples151152**Full eval:**153```154> /li-eval --corpus fixtures/voice.yaml --test fixtures/rubric.md --out reports/voice/CALIBRATION.md155[Only after those inputs, output and an actual evaluation method are authorized]156Compare every declared cell, retain actual verdict evidence, then report real counts.157If no evaluator/recorded evidence is available: NOT_EVALUATED. No calls or readiness invented.158```159160**Cell-scoped re-eval:**161```162> /li-eval --corpus fixtures/voice.yaml --test fixtures/rubric-v2.md --cells R2163Illustration: R2 now has 2/2 correct good and 2/2 correct bad.164Selected cell: PASS. Run: PARTIAL_SCOPE (1 of 3 declared cells).165R1 and R3 were not re-evaluated; no whole-corpus certification or ship clearance.166```167168**Stricter threshold:**169```170> /li-eval --threshold 95171[With the same explicitly selected inputs and real evidence]17219/20 good and 20/20 bad meet 95%; 18/20 good does not.173Report the actual scope. A mandatory threshold cannot be relaxed by this skill.174```175176## See also177178- the pack's voice corpus — the input (`resolve_pack_field voice.corpus`)179- the pack's voice test rubric — the rubric180- the calibration file — the output181- the applicable verified profile and review/control contract — separate evidence obligations182- [ADR-0021](../../../.claude/decisions/0021-eval-harness.md) — a staged general eval harness,183 not proof that this voice method already has an automated runner