jacob-balslev Evaluation
Concept of the skill
Evaluation is a gate, not a compliment: compare the requested outcome, produced artifact, evidence trail, and meaningful failure modes before deciding whether the work is done.
Coverage
This skill covers evidence-based evaluation of a completed artifact or task result:
- Skeptical critic stance: assume the work may be incomplete until evidence says otherwise.
- Goal alignment: compare the artifact to the original user request, acceptance criteria, and any stated constraints.
- Evidence inventory: name the checks, files, sources, tests, screenshots, traces, or manual inspections that support the verdict.
- 1-5 scoring: assign a numeric score with a one-sentence justification and a visible score ceiling when evidence is missing.
- Evaluation-revision loop: record required revisions, apply them, re-run focused verification, and rescore.
- Finding discipline: preserve every material gap with severity, evidence, required action, and status.
- Boundary routing: separate holistic evaluation from diff review, test planning, eval-suite design, debugging, and methodology design.
- Honest completion reporting: distinguish passed checks, unverified claims, accepted risks, deferred work, and blockers.
Philosophy of the skill
Agents are biased toward marking their own work done once the visible output looks plausible. That optimism is useful for momentum and dangerous at the finish line. Evaluation exists to slow the final decision just enough to ask: did this artifact actually satisfy the request, and can we prove it?
Modern agent-evaluation practice points in the same direction. OpenAI Evals treats evals as runnable checks for LLM systems. Anthropic frames an eval around tasks, trials, graders, traces, outcomes, and an evaluation harness, and recommends eval-driven development for agent capabilities. Google ADK separates final-response quality from trajectory and tool-use evaluation. NIST's AI Resource Center places testing, evaluation, verification, and validation inside broader AI risk-management practice. The shared lesson is simple: quality claims need observable criteria, not confidence.
This skill is the lightweight human-readable gate for a concrete deliverable. It does not replace automated eval suites or tests; it decides whether the available evidence is enough to accept the work, revise it, or block it.
Evaluation Inputs
Gather the smallest sufficient evidence packet before scoring.
| Input | What to inspect | Failure if missing |
|---|---|---|
| Original request | User goal, constraints, negative boundaries, requested output | Score may reward the wrong artifact |
| Acceptance criteria | Explicit checklist, plan, issue text, or implied done conditions | Missing requirements hide behind a high score |
| Artifact | Changed files, report, document, UI, skill, commit, or final answer | Evaluation becomes abstract opinion |
| Verification evidence | Commands run, tests, lint, screenshots, traces, source reads, links checked | Claims become unverifiable |
| Domain skills | Relevant standards loaded for the artifact type | Generic scoring misses domain-specific quality gaps |
| Residual risks | Unrun checks, known limits, deferred work, accepted tradeoffs | Final report implies more certainty than exists |
If one of these inputs does not exist, say so and apply the score ceiling in the next section.
Score Rubric
Use the full 1-5 range. A score is a decision aid, not decoration.
| Score | Verdict | Use when |
|---|---|---|
| 1 | Broken or wrong | Fails the core request, cannot be used, introduces obvious harm, or lacks the artifact being evaluated. |
| 2 | Partial or risky | Some useful work exists, but important requirements, evidence, safety checks, or domain constraints are missing. |
| 3 | Functional but incomplete | The main request is satisfied, yet polish, completeness, edge cases, documentation, or verification is weaker than the task deserves. |
| 4 | Professional and acceptable | Requirements are met, evidence is concrete, risks are handled or disclosed, and remaining issues are minor or explicitly accepted. |
| 5 | Exemplary | Goes beyond acceptance with robust edge-case handling, clear evidence, maintainable structure, excellent reporting, and no known material gaps. |
Score Ceilings
Apply ceilings mechanically. Do not negotiate them upward with confidence language.
| Condition | Maximum score |
|---|---|
| Original request or acceptance criteria were not checked | 2 |
| Verification evidence is absent or only assumed | 2 |
| Any required action remains unresolved | 3 |
| A relevant domain skill or standard was obviously needed but not consulted | 3 |
| Security, privacy, credential, customer-data, or destructive-action risk is unresolved | 2 |
| Artifact passes tests but docs, state, publication, or reporting requirements are incomplete | 3 |
| Eval, routing, freshness, or verification claims exceed evidence from this run | 2 |
| Known residual risk is real but disclosed and accepted | 4 |
Evaluation-Revision Loop
- Identify the artifact and the requested outcome in one sentence.
- Inventory evidence: changed files, relevant sources, commands, checks, screenshots, traces, or manual inspections.
- Load the domain skills or standards that govern the artifact type.
- Score each dimension: request fit, evidence sufficiency, artifact quality, risk handling, and reporting completeness.
- List every material finding with severity, evidence, required action, and status.
- Revise the artifact for every required action inside the current scope.
- Re-run focused verification that matches the revised surface.
- Rescore and produce a final accept, accept-with-risk, revise, or block verdict.
Do not mark the task done between steps 5 and 7. Findings are work, not commentary.
Finding Format
Use this shape when the evaluation finds a gap:
| Field | Required content |
|---|---|
| Severity | CRITICAL, HIGH, MEDIUM, LOW, or INFO |
| Surface | File, section, behavior, command, source, or report area |
| Evidence | Concrete line, command result, screenshot, source URL, or observed absence |
| Problem | What makes the artifact less correct, complete, safe, maintainable, or verifiable |
| Required action | What must change, or why the risk is explicitly accepted or deferred |
| Status | remediated, deferred, accepted, or open |
Preserve all findings. Prioritization may order them; it must not hide them.
Dimension Checklist
| Dimension | Questions |
|---|---|
| Request fit | Did the artifact solve the exact task, including negative instructions and scope limits? |
| Completeness | Are all required files, states, docs, artifacts, and publication steps included? |
| Correctness | Does evidence show the artifact behaves or reads correctly in the relevant cases? |
| Domain quality | Were the appropriate domain skills, standards, or source docs applied? |
| Verification | Are checks reproducible, focused, and matched to the risk surface? |
| Risk handling | Are privacy, safety, destructive action, regression, and residual uncertainty handled? |
| Reporting | Does the final report distinguish done work, evidence, residual risks, and unrun checks? |
Boundary Decisions
| If the user needs... | Use | Why |
|---|---|---|
| A grader, eval dataset, hard negatives, or pass threshold | eval-driven-development | That skill designs the eval system; this skill applies a completion score to one artifact. |
| A pull request or diff reviewed line by line | code-review | Code review owns technical findings and merge verdicts. |
| A decision about unit, integration, contract, or e2e coverage | testing-strategy | Testing strategy chooses test scope before or during implementation. |
| A process, checklist, or quality gate sequence before work starts | methodology | Methodology designs why and when gates exist. |
| Deterministic skill-library lint, manifest, drift, overlap, or routing health checks | skill-infrastructure | Skill infrastructure runs library health tooling; evaluation scores the deliverable and evidence. |
| Root-cause analysis of a known failure | debugging | Debugging investigates the cause; evaluation judges a completed result. |
Source Notes
- OpenAI Evals grounds the idea that LLM-system quality can be measured through reusable evaluation tasks and benchmark registries.
- Anthropic's agent-eval guidance grounds tasks, trials, graders, traces, outcomes, harnesses, and eval-driven development.
- Anthropic's evaluation-docs guidance supports task-specific criteria, edge cases, automation where possible, and calibrated rubric grading.
- Google ADK grounds the distinction between evaluating final responses and evaluating trajectory or tool use.
- NIST AIRC grounds evaluation inside testing, evaluation, verification, and validation for AI risk management.
Verification
After applying this skill, verify:
- The original request and acceptance criteria were explicitly checked.
- The evaluated artifact and all changed or relevant files were named.
- Verification evidence was listed with commands, outputs, source reads, traces, or inspected artifacts.
- Relevant domain skills or standards were applied, or their absence was called out.
- A 1-5 score was assigned with score ceilings respected.
- Every material finding has severity, surface, evidence, required action, and status.
- Residual risks and unrun checks are stated instead of hidden.
- Any required revision was followed by focused re-verification and rescoring.
Do NOT Use When
| Use instead | When |
|---|---|
| eval-driven-development | You are designing eval cases, rubrics, graders, hard negatives, or acceptance thresholds for future automated evaluation. |
| code-review | You are reviewing a PR, diff, or code change line by line for correctness, security, maintainability, and comments. |
| testing-strategy | You are deciding which tests to write, at which level, and with what regression target. |
| methodology | You are designing the process, checklist, or quality gate sequence before implementation exists. |
| skill-infrastructure | You are running deterministic health tooling across a skill library. |
| debugging | You are investigating why a live run, command, product behavior, or shipped change failed. |
Skill Graph context
Classification
- Subject:
ai-engineering - Public:
true - Domain:
engineering/evaluation - Scope: Evidence-based evaluation of completed agent tasks, implementations, documents, skill upgrades, and other deliverables against the original request, acceptance criteria, verification evidence, quality rubric, and residual risks before calling the work done. Portable across software, documentation, and agent-output review when the evaluator can inspect the request, artifact, and evidence. Excludes designing eval datasets or graders (eval-driven-development), line-by-line diff review (code-review), choosing test levels (testing-strategy), designing the process before work starts (methodology), and root-cause investigation of known failures (debugging).
When to use
- score this completed agent task against the original request, changed files, tests, and remaining risks before we call it done
- act as a skeptical critic and tell me whether this deliverable is actually complete or only technically working
- run a completion gate on this implementation: what is the 1-5 score, what evidence supports it, and what must be revised?
- evaluate whether this documentation update satisfies the acceptance criteria and preserves all required information
- after the focused checks pass, decide whether the skill upgrade can be marked complete without inflating eval claims
- review the final artifact and produce a scorecard with blockers, required revisions, residual risks, and verification evidence
- Triggers:
evaluation-skill,critic-loop-skill,quality-gate-skill
Not for
- design a new eval dataset, grader, and hard negatives for this router
- review this pull request line by line for bugs and security issues
- choose unit versus integration versus end-to-end tests for this feature
- design the whole implementation methodology and quality gate sequence before work starts
- debug why the existing run failed in production
- author a new SKILL.md from scratch
Related skills
- Verify with:
epistemic-grounding,code-review,best-practice,testing-strategy - Related:
methodology,debugging,best-practice,testing-strategy,code-review,epistemic-grounding,eval-driven-development
Concept
- Mental model: Evaluation is a gate, not a compliment: compare the requested outcome, produced artifact, evidence trail, and meaningful failure modes before deciding whether the work is done.
- Purpose: Prevent optimistic completion claims by making scores, evidence gaps, required revisions, and residual risks explicit before acceptance.
- Boundary: This skill applies after a concrete artifact or task result exists. It does not design eval suites, review a diff line by line, choose test levels, debug a failure, or design the process before implementation.
- Analogy: Evaluation is an inspection gate at the end of a production line: passing one measurement is not enough if the product misses the order, lacks traceability, or carries a known risk.
- Common misconception: Tests passing, lint passing, or a plausible final answer is not the same as completion. Completion requires fit to request, sufficient evidence, handled risks, and honest reporting.
Grounding
- Mode:
universal - Truth sources:
https://github.com/openai/evals,https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents,https://docs.anthropic.com/en/docs/test-and-evaluate/develop-tests,https://adk.dev/evaluate/,https://airc.nist.gov/
Keywords
evaluation,quality gate,deliverable review,completion review,evidence-based score,skeptical critic,score 1-5,evaluation revision loop,acceptance criteria review,done gate