stas2-git Agent Eval Case Builder
Use this skill to turn desired behavior and known failures into reusable eval cases.
Workflow
- Define the behavior under test and the boundary of the agent's authority.
- Collect source examples: specs, user corrections, incidents, edge cases, support tickets, traces, or known risks.
- Create a balanced eval set:
- happy path,
- edge case,
- ambiguous request,
- negative/non-trigger,
- adversarial or unsafe request,
- regression from a real failure.
- Write expected behavior and pass/fail rubric.
- Include trajectory checks for tool-using or action-allowed agents.
- Store failures as future regression cases.
References
- Read
references/eval-case-template.mdfor dataset shape and rubric patterns.
Evaluation Prompts
- Positive: "Make eval cases for this agent before we deploy it." Expected: dataset with rubrics.
- Positive edge: "Turn these user corrections into regression evals." Expected: failure-mined cases.
- Negative: "Run the unit tests." Expected: no eval-case design unless creating evals.