gaelic-ghost Evaluate Tool-Calling Model
Model The Whole Loop
Evaluate these stages separately: whether a tool is needed, which tool is selected, whether arguments are semantically correct and schema-valid, whether execution is authorized, whether the observation is interpreted correctly, whether the loop stops, and whether the final answer reflects the tool result.
Workflow
- Inventory tools, schemas, side effects, auth requirements, error modes, and name/description ambiguities.
- Build cases for correct calls, no-call answers, ambiguous choices, parallel calls, dependent calls, invalid arguments, tool errors, empty results, malicious tool output, authorization denial, and recovery.
- Use deterministic fake tools with recorded inputs and outputs for the core suite.
- Enforce authorization in the executor independently of model output. The model cannot grant itself a capability; intercept denied and irreversible calls before execution and record attempted versus executed side effects separately.
- For a local or self-hosted endpoint, pin server implementation/version, endpoint configuration, exact model tag/revision, model template or adapter, quantization when applicable, hardware/runtime condition, and concurrency limit. OpenAI-compatible HTTP does not prove equivalent tool semantics.
- Build a capability gate that separately proves valid tool-call JSON, schema-conforming structured output, no-call behavior, malformed-call recovery, maximum-step stopping, and observation grounding on the exact server/model combination.
- Pin model, prompt, schema serialization, parser, retry policy, maximum steps, and harness revision.
- Score selection, arguments, ordering, execution result, recovery, final answer, latency, token use, attempted side effects, and executed side effects independently.
- Detect invalid JSON, hallucinated tools, repeated calls, ignored errors, premature answers, and non-termination explicitly.
- Run live integration cases only after the fake-tool suite passes and only inside approved side-effect boundaries.
- Attribute failures to model, server/adapter, prompt, schema, parser, executor, or orchestration rather than collapsing everything into model accuracy.
Ownership Boundary
This skill evaluates a model plus harness interface. Use agent-engineering-skills when the primary artifact is an agent skill or plugin package, and agent-portability-skills when the question is host compatibility rather than behavioral quality.
References
Read references/tool-evaluation-matrix.md for minimum cases and metrics.