Improvement Evaluator

Improvement Evaluator

Measures whether a Skill actually makes AI perform better on real tasks, not just whether the SKILL.md document looks well-structured.

Why Execution Testing Matters

Structural scoring (word count, section presence, formatting) correlates poorly with actual AI task performance. Internal benchmarks showed R²=0.00 between document-structure scores and execution pass rates across 40+ skill evaluations. A perfectly formatted SKILL.md can still produce failing task outputs if the instructions mislead the model or omit critical constraints.

Tradeoff: Execution testing is slower and more expensive than structural checks because it invokes the AI model once per task. A 7-task suite at pass@1 costs roughly 7 API calls per candidate plus 7 for the baseline. This is acceptable because structural scoring alone gives no signal about whether the skill actually works. To offset cost, the evaluator caches baseline results for 7 days and supports --pass-k 1 (single attempt) as the default to keep runs lean.

When to Use

• Verify that a SKILL.md change improves AI task execution, not just document structure
• Run a task suite against a candidate SKILL.md and compare pass rate with baseline
• Get execution_pass_rate as a concrete quality metric for gating decisions
• Validate that a newly written task suite produces a sane baseline (>20% pass rate)
• Compare two versions of a skill on the same task suite to detect regressions
• Feed execution deltas into the improvement-gate for accept/reject decisions
• Debug low scores by inspecting per-task pass/fail details in the output artifact
• Run standalone evaluations during skill development without a full pipeline

When NOT to Use

• Checking SKILL.md structure quality only (use improvement-learner instead)
• Scoring candidates with semantic rubrics before execution (use improvement-discriminator)
• Running the full generate-score-evaluate-execute-gate pipeline (use improvement-orchestrator)
• Measuring document formatting, section counts, or word-level metrics

Task Suite Format

A task suite is a YAML file that defines what tasks to run and how to judge them. Each suite targets a specific skill and contains 5-10 tasks covering the skill's core behaviors. The schema is versioned at "1.0".

# task_suite.yaml -- minimal complete example
skill_id: "target-skill-name"
version: "1.0"
tasks:
  - id: "task-keyword-check"
    description: "Verify output mentions required concepts"
    prompt: "Given these scores {accuracy: 0.9}, what quality tier?"
    judge:
      type: "contains"
      expected: ["POWERFUL"]
    timeout_seconds: 30

  - id: "task-semantic-quality"
    description: "Rubric-scored analysis quality"
    prompt: "Accuracy dropped 0.9 to 0.8 but coverage rose. Accept?"
    judge:
      type: "llm-rubric"
      rubric: "Must mention trade-off analysis and give a recommendation"
      pass_threshold: 0.7
    timeout_seconds: 120

Validation rules enforced at load time:

• skill_id must be non-empty.
• version must equal "1.0".
• Every task needs a unique id, a non-empty prompt, and a judge block.
• Judge type must be one of contains, pytest, or llm-rubric.
• For contains: expected must be a non-empty list of strings.
• For pytest: test_file must start with fixtures/ (path-traversal guard).
• For llm-rubric: rubric must be non-empty.

See references/task-format.md and references/writing-tasks-guide.md for detailed patterns and anti-patterns.

Judge Types

The evaluator supports three judge types. Choose based on determinism needs and output complexity.

Because deterministic judges (Contains, Pytest) are fast and free while LLM judges cost an API call per evaluation, prefer deterministic judges when the pass condition can be expressed as keyword presence or structured format. Reserve LLMRubricJudge for tasks where semantic quality matters and no deterministic proxy exists.

Judge configuration examples:

# ContainsJudge -- all keywords must appear (case-insensitive)
judge:
  type: "contains"
  expected: ["validation", "sanitiz", "error handling"]

# PytestJudge -- test file receives AI output path via AI_OUTPUT_FILE
judge:
  type: "pytest"
  test_file: "fixtures/test_output_format.py"

# LLMRubricJudge -- score 0.0-1.0, pass if >= threshold
judge:
  type: "llm-rubric"
  rubric: |
    Score 0.0-1.0:
    - 0.8+: Correct analysis with actionable recommendation
    - 0.5-0.8: Partial analysis, missing specifics
    - <0.5: Generic or incorrect
  pass_threshold: 0.7

LLMRubricJudge supports --mock mode for local testing without API calls. In mock mode the judge returns a fixed passing score so you can verify the pipeline wiring without incurring cost.

Running with a suspiciously weak baseline (baseline pass rate < 20%): → Continues with a warning in evaluation.baseline_health and warnings[]. Review the suite, but the pipeline does not hard-stop.

CLI Reference

Two operating modes: pipeline mode (with ranking artifact from discriminator) and standalone mode (direct evaluation during development).

# Pipeline mode -- requires ranking artifact from discriminator stage
python3 scripts/evaluate.py \
  --input ranking-artifact.json \
  --candidate-id cand-01-docs \
  --task-suite task_suites/target-skill/task_suite.yaml \
  --state-root /tmp/eval-state \
  --pass-k 1 \
  --baseline-cache-dir /tmp/baseline-cache \
  --eval-threshold 6.0 \
  --output /tmp/eval-result.json

# Standalone mode -- evaluate a skill directly without pipeline artifacts
python3 scripts/evaluate.py \
  --standalone \
  --task-suite task_suites/deslop/task_suite.yaml \
  --skill-path ./skills/deslop \
  --state-root /tmp/eval-state \
  --mock

Output Artifacts

The evaluator writes a JSON artifact to <state-root>/evaluations/<run-id>.json (or the path specified by --output). Downstream consumers are the improvement-gate and improvement-orchestrator.

Verdict logic: pass when delta >= 0 (candidate is at least as good as baseline). skipped when candidate discriminator score is below --eval-threshold. Weak baselines no longer force error; they are recorded as warnings so execution evidence can still flow through the pipeline.

Related Skills

• improvement-discriminator -- Runs semantic scoring before this stage. Produces the ranking artifact that this evaluator consumes. Use discriminator when you need LLM panel review scores, not execution-based pass rates.
• improvement-gate -- Consumes this evaluator's output artifact. Applies a 6-layer mechanical gate (Schema, Compile, Lint, Regression, Review, HumanReview) to decide whether to accept or reject the change.
• improvement-orchestrator -- Coordinates the full pipeline: generate, discriminate, evaluate, execute, gate. Use orchestrator when you want the end-to-end flow rather than running individual stages.
• improvement-learner -- Structural quality scoring (6-dimension). Use learner when you only care about document quality metrics, not execution effectiveness.