Tool Quality Audit
Overview
Evaluate a tool’s reliability with a checklist and small smoke tests. Emphasize deterministic outputs, clear errors, and stable schemas.
Quick start
- Fill
templates/tool_audit.json. - Run smoke tests manually or via your harness.
- Record findings in
results.json.
Core Guidance
- Prefer deterministic checks before LLM-based grading.
- Verify error contracts (consistent codes/messages).
- Validate schemas are stable and documented.
- Record latency and failure modes.
Resources
references/tool-audit-checklist.md: Reliability and contract checklist.templates/tool_audit.json: Audit scaffold for a tool.
Validation
- Ensure audit file is filled and failures are actionable.