Evals Run Skill
[!IMPORTANT]
Workflow skill for evals run.
Optional args: slug=, ticket=<id/url>, mode=interactive|autonomous|channel, channel=, auto_continue=true|false, profile=business|hybrid|technical.
Instructions
When the user asks to perform this workflow, execute the following steps:
description: Run blinded live skill evals and publish reproducible v2 results.
Goal
Measure whether a skill changes agent behavior with isolated, immutable, outcome-based eval evidence. Distinguish textual transcript assertions from executable task verifications.
Steps
1. Choose or resume a run
For ordinary maintenance after a complete catalog baseline exists, run pnpm evals:baseline first. It creates or resumes a selective manifest, reuses only compatible evidence, and prints the model, reasoning level, concurrency, and fresh-answer count without starting workers.
Review that plan before spending quota. Start workers only with pnpm evals:baseline -- --execute; the default is gpt-5.6-luna with high reasoning and one worker. Override intentionally with EVALS_MODEL, EVALS_REASONING_EFFORT, or EVALS_CONCURRENCY (maximum four workers).
If usage is exhausted, keep the run directory and rerun the identical --execute command after access resumes; completed answers are reused automatically.
Use pnpm evals:manifest -- --category <category> for one category or pnpm evals:manifest -- --all for the complete catalog.
Use pnpm evals:manifest -- --resume <runId> only when deliberately continuing an existing run; a new invocation always creates a collision-safe run ID.
Record the printed run ID. The manifest records source hashes, the v2 schema, and the generation protocol.
2. Answer each blinded case
- Run each baseline and with-skill arm in a separate worker/context.
- Baseline receives only the prompt. With-skill receives the same prompt plus that skill's
SKILL.md.
- Neither arm receives coaching pressure or answer-anchor hints. Textual assertions measure transcript evidence; distinguish clearly from executable task outcomes.
- Trigger cases receive only the skill name and one-line description; never open the full skill body or expose the expected label.
- Trigger prompt filenames use opaque case IDs; never infer the expected label from filenames or ordering.
- For
all runs, write answers under answers/<category>/<skill>/<case>; category runs use answers/<skill>/<case>.
- Mark known compromised baselines in the manifest and do not use them for delta calculations until clean reruns replace them.
3. Complete and score
- Fill
metadata.agent, metadata.model, and metadata.completedAt after every required answer exists.
- Run
pnpm evals:score -- --run <runId>.
- Scoring refuses to write
results.json while any arm is pending, verifies source hashes, and writes one immutable inputs.json snapshot before publishing v2 results.
4. Report and verify
- Run
pnpm evals:report to project aggregate runs into the newest complete category partitions and update physical history/archive records.
- Run
pnpm evals:verify -- --run <runId> and, before handoff, pnpm evals:verify -- --all.
- Confirm case pass rate, assertion pass rate, trigger recall, trigger specificity, and balanced trigger accuracy. Treat baseline and delta as
n/a for compromised arms.
- Never hand-edit
results.json, transcripts, history, or archives. Fix inputs or eval definitions and regenerate.
Output
Run Summary
Evidence
Known Risks
Outcome Report
feature_status: implemented | partially_implemented | blocked
requirement_trace: manifest -> inputs -> results -> report -> verification
completed_evidence: []
missing_evidence: []
decision_needed: []
recommended_next_workflow: verify-work
1---2name: evals-run3description: "Workflow skill for evals run."4metadata:5 source: https://skillmd.com/skills/hoangnguyen0403/evals-run6 internal: true7 triggers:8 keywords:9 - evals run10 - workflow11---1213# Evals Run Skill1415> [!IMPORTANT]16> Workflow skill for evals run.1718Optional args: slug=<feature>, ticket=<id/url>, mode=interactive|autonomous|channel, channel=<id>, auto_continue=true|false, profile=business|hybrid|technical.1920## Instructions2122When the user asks to perform this workflow, execute the following steps:2324description: Run blinded live skill evals and publish reproducible v2 results.2526# Goal2728Measure whether a skill changes agent behavior with isolated, immutable, outcome-based eval evidence. Distinguish textual transcript assertions from executable task verifications.29# Steps3031## 1. Choose or resume a run3233- For ordinary maintenance after a complete catalog baseline exists, run `pnpm evals:baseline` first. It creates or resumes a selective manifest, reuses only compatible evidence, and prints the model, reasoning level, concurrency, and fresh-answer count without starting workers.34- Review that plan before spending quota. Start workers only with `pnpm evals:baseline -- --execute`; the default is `gpt-5.6-luna` with `high` reasoning and one worker. Override intentionally with `EVALS_MODEL`, `EVALS_REASONING_EFFORT`, or `EVALS_CONCURRENCY` (maximum four workers).35- If usage is exhausted, keep the run directory and rerun the identical `--execute` command after access resumes; completed answers are reused automatically.3637- Use `pnpm evals:manifest -- --category <category>` for one category or `pnpm evals:manifest -- --all` for the complete catalog.38- Use `pnpm evals:manifest -- --resume <runId>` only when deliberately continuing an existing run; a new invocation always creates a collision-safe run ID.39- Record the printed run ID. The manifest records source hashes, the v2 schema, and the generation protocol.4041## 2. Answer each blinded case4243- Run each baseline and with-skill arm in a separate worker/context.44- Baseline receives only the prompt. With-skill receives the same prompt plus that skill's `SKILL.md`.45- Neither arm receives coaching pressure or answer-anchor hints. Textual assertions measure transcript evidence; distinguish clearly from executable task outcomes.46- Trigger cases receive only the skill name and one-line description; never open the full skill body or expose the expected label.47- Trigger prompt filenames use opaque case IDs; never infer the expected label from filenames or ordering.48- For `all` runs, write answers under `answers/<category>/<skill>/<case>`; category runs use `answers/<skill>/<case>`.49- Mark known compromised baselines in the manifest and do not use them for delta calculations until clean reruns replace them.50## 3. Complete and score5152- Fill `metadata.agent`, `metadata.model`, and `metadata.completedAt` after every required answer exists.53- Run `pnpm evals:score -- --run <runId>`.54- Scoring refuses to write `results.json` while any arm is pending, verifies source hashes, and writes one immutable `inputs.json` snapshot before publishing v2 results.5556## 4. Report and verify5758- Run `pnpm evals:report` to project aggregate runs into the newest complete category partitions and update physical history/archive records.59- Run `pnpm evals:verify -- --run <runId>` and, before handoff, `pnpm evals:verify -- --all`.60- Confirm case pass rate, assertion pass rate, trigger recall, trigger specificity, and balanced trigger accuracy. Treat baseline and delta as `n/a` for compromised arms.61- Never hand-edit `results.json`, transcripts, history, or archives. Fix inputs or eval definitions and regenerate.6263# Output6465## Run Summary6667## Evidence6869## Known Risks7071## Outcome Report7273feature_status: implemented | partially_implemented | blocked74requirement_trace: manifest -> inputs -> results -> report -> verification75completed_evidence: []76missing_evidence: []77decision_needed: []78recommended_next_workflow: verify-work79