Bounded Evaluation
Use this skill when a change needs a small, explicit evaluation gate.
Use When
- reviewing skill changes, AGENTS.md rules, prompt playbooks, or SET bundle changes;
- comparing two agent outputs or proposals;
- using an LLM judge where bias or weak rubrics could mislead;
- creating activation tests, regression cases, or ship gates for agent behavior;
- deciding whether a repeated workflow improved after a bounded edit.
Evaluation Contract
A bounded eval must state:
- target artifact or behavior;
- candidate outputs or versions;
- rubric dimensions and weights, if any;
- pass/fail threshold;
- fixtures or scenarios;
- judge type: deterministic check, human review, LLM judge, or hybrid;
- known bias risks and mitigation.
Pairwise And Judge Rules
- Compare A/B outputs against the same rubric.
- Swap candidate order when using an LLM judge to detect position bias.
- Require reasons before scores, but treat reasons as evidence to inspect, not truth.
- Mark the eval invalid when swapped order changes the winner without a defensible reason.
- Prefer deterministic checks for syntax, schema, file existence, and command success.
- Keep human approval for high-stakes or ambiguous decisions.
Workflow
- Define the smallest behavior being evaluated.
- Pick 1-5 representative fixtures or scenarios.
- Choose deterministic checks before LLM judging.
- If using pairwise judging, run both candidate orders.
- Record results, invalidations, and decision.
- Feed accepted/rejected outcomes back into the relevant ledger.
Output Shape
Use:
- eval target;
- fixtures;
- rubric;
- judge type;
- results;
- bias checks;
- decision:
accept, reject, revise, or invalid;
- follow-up artifact or ledger update.
Guardrails
- Do not publish benchmark claims without committed fixtures, arms, model, host, run count, and captured artifacts.
- Do not let an LLM judge approve its own rubric changes.
- Do not use pairwise preference as proof of correctness.
- Do not hide invalid or inconsistent judge results.
Provenance
Adapted from advanced-evaluation patterns in muratcankoylan/Agent-Skills-for-Context-Engineering, aligned with ABVX SkillOpt validation gates.
1---2name: bounded-evaluation3description: Design small evaluation gates for agent outputs, skill changes, prompts, and tool contracts. Use when a change needs rubric checks, pairwise comparison, position-bias mitigation, regression fixtures, or LLM-as-judge discipline without building a heavy eval platform.4license: MIT5---67# Bounded Evaluation89Use this skill when a change needs a small, explicit evaluation gate.1011## Use When1213- reviewing skill changes, AGENTS.md rules, prompt playbooks, or SET bundle changes;14- comparing two agent outputs or proposals;15- using an LLM judge where bias or weak rubrics could mislead;16- creating activation tests, regression cases, or ship gates for agent behavior;17- deciding whether a repeated workflow improved after a bounded edit.1819## Evaluation Contract2021A bounded eval must state:2223- target artifact or behavior;24- candidate outputs or versions;25- rubric dimensions and weights, if any;26- pass/fail threshold;27- fixtures or scenarios;28- judge type: deterministic check, human review, LLM judge, or hybrid;29- known bias risks and mitigation.3031## Pairwise And Judge Rules3233- Compare A/B outputs against the same rubric.34- Swap candidate order when using an LLM judge to detect position bias.35- Require reasons before scores, but treat reasons as evidence to inspect, not truth.36- Mark the eval invalid when swapped order changes the winner without a defensible reason.37- Prefer deterministic checks for syntax, schema, file existence, and command success.38- Keep human approval for high-stakes or ambiguous decisions.3940## Workflow41421. Define the smallest behavior being evaluated.432. Pick 1-5 representative fixtures or scenarios.443. Choose deterministic checks before LLM judging.454. If using pairwise judging, run both candidate orders.465. Record results, invalidations, and decision.476. Feed accepted/rejected outcomes back into the relevant ledger.4849## Output Shape5051Use:5253- eval target;54- fixtures;55- rubric;56- judge type;57- results;58- bias checks;59- decision: `accept`, `reject`, `revise`, or `invalid`;60- follow-up artifact or ledger update.6162## Guardrails6364- Do not publish benchmark claims without committed fixtures, arms, model, host, run count, and captured artifacts.65- Do not let an LLM judge approve its own rubric changes.66- Do not use pairwise preference as proof of correctness.67- Do not hide invalid or inconsistent judge results.6869## Provenance7071Adapted from advanced-evaluation patterns in `muratcankoylan/Agent-Skills-for-Context-Engineering`, aligned with ABVX SkillOpt validation gates.
markoblogo