stas2-git Agent Observability Trace Review
Use this skill to judge how an agent behaved, not just what it answered.
Workflow
- Reconstruct the run: user request, loaded context, tools, actions, outputs, approvals, and final response.
- Identify the intended success criteria.
- Inspect trajectory for:
- wrong tool or missing tool,
- untrusted content treated as instruction,
- unnecessary authority,
- loops/retries/cost spikes,
- missing verification,
- unsafe side effects,
- drift from the latest user request.
- Compare final answer to evidence in the trace.
- Produce findings and recommended harness/eval fixes.
References
- Read
references/trace-rubric.mdfor review dimensions and output template.
Evaluation Prompts
- Positive: "Review this agent transcript and tell me where it went wrong." Expected: trajectory findings.
- Positive edge: "The final answer was correct; was the run safe?" Expected: inspect tool path and approvals.
- Negative: "Summarize this normal meeting transcript." Expected: no trace review unless it is an agent run.