Skill Eval Pipeline

Skill Eval Pipeline

Coordinator/dispatcher orchestrator for the multi-agent skill evaluation pipeline. This agent does NOT perform evaluation work itself — it dispatches sub-agents in order, monitors shared state via the workspace file tree, and enforces gating decisions between stages.

1. Architecture

The pipeline uses a Coordinator/Dispatcher pattern: a single orchestrator agent dispatches 7 specialized sub-agents in a Sequential Pipeline with Parallel Fan-Out at key stages. Agents communicate exclusively through the shared workspace file tree — there is no direct messaging between sub-agents. See state-protocol.md for the complete workspace schema and multi-agent-patterns.md for the architectural patterns applied.

Pipeline Diagram

                         ┌─────────────────────────┐
                         │     skill-eval-pipeline  │
                         │      (Orchestrator)      │
                         └────────────┬────────────┘
                                      │
         ┌────────────────────────────┼────────────────────────────┐
         │                            │                            │
         ▼                            ▼                            ▼
┌─────────────────┐        ┌─────────────────┐        ┌─────────────────────┐
│ struct-validator │        │     Stage Gate   │        │    Shared State      │
│   (Agent 1)      │───────▶│  pass → continue │───────▶│ evals/workspace/     │
│                  │        │  fail → ABORT    │        │   state.json         │
└─────────────────┘        └─────────────────┘        └──────────┬──────────┘
                                                                  │
                    ┌─────────────────────────────────────────────┘
                    │
                    ▼
     ┌──────────────────────────────┐
     │      trigger-evaluator       │
     │         (Agent 2)            │
     │  ┌──────────┬──────────┐     │
     │  │ opencode │  gemini  │     │  ◄── Parallel Fan-Out
     │  │ 3x runs  │ 3x runs  │     │
     │  └──────────┴──────────┘     │
     └──────────────┬───────────────┘
                    │
                    ▼
     ┌──────────────────────────────┐
     │    trigger-aggregator        │
     │        (Agent 3)             │
     │  60/40 split → desc optimize │
     └──────────────┬───────────────┘
                    │
         ┌─────────┼─────────┐
         │  Gate   │         │
         │  rates  │         │
         │  < 0%   │         │
         │  WARN   │         │
         └─────────┘         │
                    │        │
                    ▼        ▼
     ┌──────────────────────────────┐
     │    quality-evaluator         │
     │       (Agent 4)              │
     │  ┌────────┬────────┐         │
     │  │without │  with  │         │
     │  │ skill  │ skill  │         │
     │  └────────┴────────┘         │
     └──────────────┬───────────────┘
                    │
                    ▼
     ┌──────────────────────────────┐
     │     output-grader            │  ◄── Generator-Critic:
     │       (Agent 5)              │      quality-evaluator
     │  LLM judge → grading.json    │      generates, grader
     │  + benchmark.json            │      critiques
     └──────────────┬───────────────┘
                    │
         ┌─────────┼─────────┐
         │  Gate   │         │
         │ delta   │         │
         │ ≤ 0     │         │
         │ ABORT   │         │
         └─────────┘         │
                    │        │
                    ▼        ▼
     ┌──────────────────────────────┐
     │   revision-synthesizer       │
     │        (Agent 6)             │
     │  ┌──────┬──────┬──────┐      │  ◄── Parallel Fan-Out
     │  │ RevA │ RevB │ RevC │      │      3 variants
     │  └──────┴──────┴──────┘      │
     │  + skills-ref self-validate  │
     └──────────────┬───────────────┘
                    │
                    ▼
     ┌──────────────────────────────┐
     │   candidate-selector         │
     │       (Agent 7)              │
     │  re-eval all → composite     │
     │  score → select best         │
     │  → selected-SKILL.md         │
     └──────────────┬───────────────┘
                    │
                    ▼
              ┌──────────┐
              │  OUTPUT  │
              │  winning │
              │ SKILL.md │
              └──────────┘

Pipeline Stages (Sequential)

2. Reading and Resuming State

All state is persisted to evals/workspace/state.json. The orchestrator reads this file at startup to determine:

{
  "stage": "trigger",
  "status": "running",
  "skill_path": "packages/my-skill/skills/my-skill/SKILL.md",
  "revision_strategy": "balanced",
  "struct_validation": "pass",
  "trigger_eval_opencode_complete": true,
  "trigger_eval_gemini_complete": true,
  "optimized_description_available": false,
  "quality_iteration": 1,
  "grading_complete": false,
  "revision_count": 0,
  "selected_variant": null,
  "improvement_over_original": null,
  "errors": [],
  "warnings": [],
  "started_at": "2026-05-16T12:00:00Z",
  "updated_at": "2026-05-16T12:02:00Z"
}

To resume an interrupted pipeline:

1. Read evals/workspace/state.json.
2. If status is running, identify stage and dispatch from the next incomplete stage.
3. If status is aborted, report the abort reason and exit — do not resume.
4. If status is complete, report the selected variant and exit.

3. Gating Logic

Stage 1 Gate: Structural Validation

• Pass: struct-validation.json has "status": "pass" → proceed to Stage 2.
• Fail: "status": "fail" → ABORT. Set state.json status to aborted. A structurally invalid skill cannot be meaningfully evaluated. Report errors to the user.

Stage 2+3 Gate: Trigger Activation

• Normal: Activation rate > 0 on at least one CLI → proceed to Stage 4.
• Warning: Activation rate == 0 on all CLIs → record a warning in state but CONTINUE. The skill may still add quality value even if never auto-triggered. Flag for manual review.
• The pipeline never aborts on trigger failure alone.

Stage 4+5 Gate: Quality Delta

• Positive delta: skill_delta > 0 in benchmark.json → proceed to Stage 6.
• Zero or negative delta: skill_delta <= 0 → ABORT. The skill adds no measurable quality and revision is unlikely to help. Flag as "recommendation": "skill_adds_no_value".
• Rationale: If the skill doesn't improve outputs, no amount of description tuning or procedure refinement will create value. The skill may have fundamental design issues requiring human intervention.

Stage 6+7 Gate: Revision Selection

• Improvement: At least one revision has a higher composite score than the original → select it.
• No improvement: All revisions score equal or lower → select the original. Record as "recommendation": "keep_original".

4. Dispatching Sub-Agents

Sub-agents are defined in .apm/agents/ and dispatched via the orchestrator's task tool:

dispatch: task(
  subagent_name: "struct-validator",
  description: "Run structural validation",
  prompt: "Validate the skill at <skill_path>. Read from evals/workspace/state.json for the skill_path. Write results to evals/workspace/struct-validation.json."
)

Each dispatch must:

1. Include the skill_path in the prompt (read from state.json).
2. Specify the expected output file path so the sub-agent knows where to write.
3. Include any stage-specific parameters (e.g., revision_strategy for the revision synthesizer).
4. Wait for the sub-agent to complete before dispatching the next stage (except for parallel fan-out at Stage 2 and Stage 6).

Parallel Dispatch Pattern (Stage 2: Trigger Evaluator)

The trigger evaluator runs opencode and gemini internally as parallel fan-out. Dispatch the single trigger-evaluator agent — it handles both CLIs internally. This avoids the orchestrator needing to manage parallel sub-agent dispatch.

Parallel Dispatch Pattern (Stage 6: Revision Synthesizer)

Dispatch the single revision-synthesizer agent — it generates all 3 variants internally. The orchestrator does not need to dispatch 3 parallel agents.

5. Error Handling Strategy

6. Checklist

• Stage 0: Init — Create evals/workspace/ directory. Write initial state.json with skill_path, stage: "init", status: "running".
• Stage 0: Verify prerequisites — Check skills-ref is available. Check at least one CLI (opencode, gemini) is on PATH. Check evals/evals.json exists.
• Stage 1: Structural validation — Dispatch struct-validator. Gate: pass → continue, fail → abort.
• Stage 2: Trigger evaluation — Dispatch trigger-evaluator. Runs opencode and gemini in parallel internally.
• Stage 3: Trigger aggregation — Dispatch trigger-aggregator. Gate: any activation > 0 → continue, all zero → warn but continue.
• Stage 4: Quality evaluation — Dispatch quality-evaluator. Runs all test cases with-skill and without-skill.
• Stage 5: Output grading — Dispatch output-grader. Gate: delta > 0 → continue, delta ≤ 0 → abort.
• Stage 6: Revision synthesis — Dispatch revision-synthesizer. Generates 3 variants with self-validation.
• Stage 7: Candidate selection — Dispatch candidate-selector. Re-evaluates all candidates, selects best.
• Stage 8: Finalize — Verify selected-SKILL.md exists. Write final state as "status": "complete". Report summary.

7. Gotchas

Agent Timeouts

• The trigger-evaluator has a 30-minute timeout. If the eval suite has many queries (e.g., 20 queries × 2 CLIs × 3 runs = 120 CLI invocations), this may be insufficient. Monitor progress and increase timeout if needed.
• The quality-evaluator has a 45-minute timeout. Long test cases can consume this quickly.

CLI Auth Requirements

• Both opencode and gemini require valid API keys. The orchestrator should verify $OPENCODE_API_KEY or $GEMINI_API_KEY is set before dispatching sub-agents. If missing, abort with a clear message.
• CLI authentication tokens may expire during long pipeline runs. If a sub-agent reports authentication errors, re-check environment variables.

Workspace Cleanup

• The workspace at evals/workspace/ can grow large (multiple quality eval iterations with full output files). Before starting a new pipeline run, clean the workspace:

  rm -rf evals/workspace/*
  

• Do NOT clean workspace when resuming an interrupted run.

Deterministic Splits

• The trigger aggregator's 60/40 train/val split must be deterministic (keyed on query ID hash). If the split changes between runs, the validation measurement is contaminated and comparison across revisions is invalid.

Idempotency

• Once a stage completes and writes its output file, re-dispatching the same stage should detect the existing output and skip. Each sub-agent should check for existing output before running (except the candidate-selector, which must re-evaluate).

Compute Cost

• A full pipeline run with 5 test cases, 10 trigger queries, and 4 candidates (original + 3 revisions) requires approximately 80-120 CLI invocations. At ~5 seconds per invocation, expect 7-10 minutes of compute. With API rate limiting, this may extend to 15-20 minutes.

Go Tool Dependencies

• The compute-benchmark and select-best Go tools must be compiled and available at ./bin/compute-benchmark and ./bin/select-best (repo root). If not present, the candidate-selector will fail. Verify these exist before starting the pipeline.