Capture Eval Issues

Capture noteworthy review violations for the eval framework. Use when validator-run finds review failures — judges violations and saves notable ones to evals/inventory.yml.

codagent-ai c8c8adf 3 files · 8.2 KB Updated

File contents

/capture-eval-issues

Capture noteworthy review violations into the eval inventory.

Input

This skill receives one or more review JSON file paths as arguments (space-separated). Example: /capture-eval-issues validator_logs/review_src_claude@1.0.json validator_logs/review_src_gemini@2.0.json

Procedure

  1. Read judge-prompt.md from this skill's directory
  2. Spawn a subagent to judge and capture violations:
    • Task tool: Task with subagent_type="general-purpose", model="sonnet", prompt= judge-prompt content + "\n\nReview JSON files: <file paths>\n\nSkill directory: <this skill's directory path>"
    • The subagent handles everything: reading files, judging, calling the append script
  3. Report the subagent's capture summary (the CAPTURED: line)

codagent-ai/agent-validator/tree/main/.agents/skills/capture-eval-issues commit c8c8adf4ce

Frequently asked questions

npx skillmds@latest add codagent-ai/capture-eval-issues