DRAFT Tool-Trust Verifier
Overview
Tool discovery that depends on descriptions has a failure mode the chapter names
directly: providers optimize descriptions for DISCOVERY, not accuracy. "Most
effective solution." "Trusted by Fortune 500." "Industry-leading performance."
When every tool claims to be the best, keyword-gamed descriptions defeat the
retrieval algorithms — surfacing the best marketers rather than the best tools.
At scale (thousands of tools) you cannot manually verify claims.
The chapter's answer is verification-based trust, on two mechanisms:
- Structured, verifiable capabilities. A tool does not get to claim it
"analyzes customer sentiment with unparalleled accuracy." It declares the
capability
sentiment_analysis with specific input and output types that can
be tested.
- Performance-based trust scores. Every tool begins neutral. Successful
executions raise trust; failures, high latency, or degradations lower it. The
orchestrator learns to prioritize tools that are consistently reliable.
Baidu's DRAFT (Documentation Refinement through Automated Feedback and
Testing) operationalizes this as a continuous learning loop that mirrors how a
developer learns a new API:
- Experience Gathering — an explorer probes tool boundaries, seeks edge
cases, maps failure modes, and enforces diversity to avoid redundant tests.
- Learning from Experience — analyze the gap between documentation and
reality (claims "any text input" but fails on Unicode; undocumented
payload-size latency). Systematic discovery of true capabilities, not error
logging.
- Documentation Rewriting — generate an AI-optimized spec reflecting the
discovered reality: parameter types, ranges, error conditions, real
performance. Iterate until the doc converges with actual behavior.
DRAFT sidesteps the trust problem: why worry about providers gaming descriptions
when your system discovers the truth anyway? This parallels Writer's gateway,
which rewrites descriptions preemptively (before deployment) rather than
iteratively (after observing failures) — both treat tool descriptions as an
active interface, not static metadata.
When to Use
- A tool registry ingests provider-authored or third-party descriptions
- Retrieval keeps surfacing "best-marketed" tools that then underperform
- You want a trust score to rank functionally-equivalent tools by reliability
- You are onboarding a new tool and its documentation is human-friendly prose,
not an agent-parseable spec
Phrases that invoke this skill: "verify what this tool does", "the tool
description is gamed", "trust score", "DRAFT", "rewrite the tool docs",
"doc vs reality".
When NOT to Use
- Tools you authored and fully control — you already know the true behavior;
write the structured spec directly.
- As a functional test framework. DRAFT discovers doc-vs-reality gaps
(constraints, error conditions, performance); it does not assert business
correctness. Keep your unit/integration tests.
- As a security scanner. It measures capability and reliability, not
vulnerability. Pair with
agent-shield / SkillSpector for security.
- When there is no gap to find — a tool whose docs already match its behavior
converges immediately; DRAFT adds nothing.
Process
| Step |
Input |
Action |
Output |
Verification |
| 1 |
Tool JSON (claimed_description + declared_capabilities + behavior) |
lib.load_tool_under_test(path) |
Tool dict |
Has claimed_description and a behavior model |
| 2 |
Tool dict |
lib.verify_claims(tool) |
Marketing-phrase flags + structured-capability count |
Gamed phrases flagged; verifiable True only if every capability has name+input_types+output_types |
| 3 |
Execution history |
lib.TrustScore().record(success, latency_ms) per call |
Evolving trust score |
Neutral 0.5 start; successes up, failures + slow calls down |
| 4 |
Tool + probe set |
lib.gather_experience(tool, probes) |
Observations (success/latency/error) |
Duplicate probes deduped (diversity enforced) |
| 5 |
Tool + observations |
lib.learn_from_experience(tool, obs) |
Doc-vs-reality gap report |
Discovered error conditions + gap_over_promises_any_text |
| 6 |
Tool + learning |
lib.rewrite_documentation(tool, learning) |
AI-optimized refined spec |
Refined spec lists real constraints + performance, drops marketing |
| 7 |
Tool + probes |
lib.run_draft(tool, probes) |
Full gather→learn→rewrite result |
Refined spec's error_conditions match discovered failures |
Rationalizations
| Agent rationalization |
Documented rebuttal |
| "The tool description says it handles any text input — trust it." |
DRAFT's worked example is exactly this: a tool claims "any text input" but fails on Unicode. Descriptions are gamed for discovery; probe the boundary before trusting the claim. |
| "Free-text capability descriptions are fine." |
The chapter requires structured, enumerable capabilities with input/output types precisely because free text cannot be tested. verify_claims flags a capability with null input_types as unverifiable. |
| "One trust score for the tool is enough — reliability is fixed." |
Trust is performance-based and evolves: successes up, failures and slow calls down. A tool that degrades (technical debt, maintenance windows) must lose trust over time, not keep a stale score. |
| "Just log the errors when the tool fails." |
Learning-from-experience is systematic discovery of TRUE capabilities, not error logging. It converts observed failures into refined constraints (max length, ASCII-only) that reshape future selection. |
| "Rewriting docs is the provider's job." |
Raw provider docs are written for humans and cause tool-calling errors when used as-is (Writer's finding). The rewrite — preemptive or iterative — is what makes the description an accurate agent interface. |
Red Flags
- A capability with
input_types: null marked verifiable. The structured-
capability check is broken; unverifiable free-text is passing.
- DRAFT reports
converged: true on a tool with known undocumented limits.
The probe set is not diverse enough — it never hit the boundary. Add edge-case
probes (empty, oversized, non-ASCII).
- Trust score never moves.
record is not being called on real execution
outcomes; the score is decorative.
- Refined spec still contains marketing language. The rewrite copied the
claim instead of assembling from observations.
- CLI
--help exits non-zero. SKILL.md / CLI mismatch; multi-harness invariant broken.
Non-Negotiable Verification
Run the benchmark battery.
python cli.py benchmark
Confirm marketing phrases are flagged, capabilities are NOT fully verifiable
(the sample's second capability has null input_types), the three undocumented
constraints are discovered, and a flaky-tool history drives trust below 0.5.
Prove the gap is discovered, not assumed.
python cli.py learn
The claim says "any text input"; the learning report must show
gap_over_promises_any_text: True with the discovered error conditions.
Prove the refined spec drops marketing.
python cli.py rewrite
The refined description states real constraints and performance; it contains
none of the flagged marketing phrases.
JSON round-trips.
python cli.py draft --json | python -c "import json,sys; json.load(sys.stdin)"
Security Posture
- Prompt injection. A gamed description can embed instructions. This skill
never trusts the description for behavior — it probes the tool. The
# TODO(production): seam in TrustScore.record marks where SLO-weighted
penalties and observation decay go; the explorer's probe issuance is the
production seam where real (sandboxed) calls replace _simulate.
- Data exfiltration. No network calls in
lib.py; behavior is simulated
from declarative rules. In production, run the explorer's probes in a
sandbox so boundary-probing cannot trigger real side effects.
- Privilege escalation. DRAFT probes should be read-only / idempotent —
boundary exploration must not invoke destructive tool operations. Gate probe
generation behind the same sensitive-action policy as
information-flow-control-gate.
Composition
- Feeds
skill-quality-evaluator and retrieval: the DRAFT-refined
description improves search accuracy, and the trust score is a ranking
signal for functionally-equivalent tools.
- Feeds
hierarchical-orchestration-router functional clustering — DRAFT-
refined representations are what the clusterer embeds ("embed tools based on
what they do, not what they claim").
- Pairs with
information-flow-control-gate — DRAFT learns what a tool does;
IFC governs what data may flow into it.
- Parallels the MCP Gateway's preemptive description rewrite (Palmyra X5);
DRAFT is the iterative, post-observation form of the same insight.
Source Attribution
Distilled from Agentic GraphRAG (O'Reilly), Chapter 6 — Tool Orchestration,
sections "The Trust Problem: When Tools Game the System", "Verification-based
trust", and "DRAFT: Learning what tools actually do". Named references:
- Baidu AI Search Paradigm — DRAFT (Documentation Refinement through Automated
Feedback and Testing); three-phase loop
- Writer enterprise MCP gateway — preemptive description rewrite (Palmyra X5),
the same insight applied before deployment rather than after
1---2name: draft-tool-trust-verifier3description: Establish trust in a tool by verification, not by its self-description. Flags marketing-gamed tool descriptions ("industry-leading", "trusted by Fortune 500"), requires structured testable capabilities instead of free-text claims, tracks a performance-based trust score (neutral start, successes up, failures and slow calls down), and runs the DRAFT loop — gather boundary-probing experience, learn the gap between documentation and reality, rewrite an AI-optimized spec — until the doc converges with actual behavior. Use when a tool registry ingests third-party or provider-authored descriptions that may be optimized for discovery over accuracy. NOT for tools you authored and fully control, NOT a functional test framework (it discovers doc-vs-reality gaps, it does not assert business correctness), NOT a security scanner.4---56# DRAFT Tool-Trust Verifier78## Overview910Tool discovery that depends on descriptions has a failure mode the chapter names11directly: providers optimize descriptions for DISCOVERY, not accuracy. "Most12effective solution." "Trusted by Fortune 500." "Industry-leading performance."13When every tool claims to be the best, keyword-gamed descriptions defeat the14retrieval algorithms — surfacing the best marketers rather than the best tools.15At scale (thousands of tools) you cannot manually verify claims.1617The chapter's answer is verification-based trust, on two mechanisms:18191. **Structured, verifiable capabilities.** A tool does not get to claim it20 "analyzes customer sentiment with unparalleled accuracy." It declares the21 capability `sentiment_analysis` with specific input and output types that can22 be tested.232. **Performance-based trust scores.** Every tool begins neutral. Successful24 executions raise trust; failures, high latency, or degradations lower it. The25 orchestrator learns to prioritize tools that are consistently reliable.2627Baidu's **DRAFT** (Documentation Refinement through Automated Feedback and28Testing) operationalizes this as a continuous learning loop that mirrors how a29developer learns a new API:3031- **Experience Gathering** — an explorer probes tool boundaries, seeks edge32 cases, maps failure modes, and enforces diversity to avoid redundant tests.33- **Learning from Experience** — analyze the gap between documentation and34 reality (claims "any text input" but fails on Unicode; undocumented35 payload-size latency). Systematic discovery of true capabilities, not error36 logging.37- **Documentation Rewriting** — generate an AI-optimized spec reflecting the38 discovered reality: parameter types, ranges, error conditions, real39 performance. Iterate until the doc converges with actual behavior.4041DRAFT sidesteps the trust problem: why worry about providers gaming descriptions42when your system discovers the truth anyway? This parallels Writer's gateway,43which rewrites descriptions preemptively (before deployment) rather than44iteratively (after observing failures) — both treat tool descriptions as an45active interface, not static metadata.4647## When to Use4849- A tool registry ingests provider-authored or third-party descriptions50- Retrieval keeps surfacing "best-marketed" tools that then underperform51- You want a trust score to rank functionally-equivalent tools by reliability52- You are onboarding a new tool and its documentation is human-friendly prose,53 not an agent-parseable spec5455Phrases that invoke this skill: "verify what this tool does", "the tool56description is gamed", "trust score", "DRAFT", "rewrite the tool docs",57"doc vs reality".5859## When NOT to Use6061- **Tools you authored and fully control** — you already know the true behavior;62 write the structured spec directly.63- **As a functional test framework.** DRAFT discovers doc-vs-reality gaps64 (constraints, error conditions, performance); it does not assert business65 correctness. Keep your unit/integration tests.66- **As a security scanner.** It measures capability and reliability, not67 vulnerability. Pair with `agent-shield` / SkillSpector for security.68- **When there is no gap to find** — a tool whose docs already match its behavior69 converges immediately; DRAFT adds nothing.7071## Process7273| Step | Input | Action | Output | Verification |74|------|-------|--------|--------|--------------|75| 1 | Tool JSON (claimed_description + declared_capabilities + behavior) | `lib.load_tool_under_test(path)` | Tool dict | Has `claimed_description` and a `behavior` model |76| 2 | Tool dict | `lib.verify_claims(tool)` | Marketing-phrase flags + structured-capability count | Gamed phrases flagged; `verifiable` True only if every capability has name+input_types+output_types |77| 3 | Execution history | `lib.TrustScore().record(success, latency_ms)` per call | Evolving trust score | Neutral 0.5 start; successes up, failures + slow calls down |78| 4 | Tool + probe set | `lib.gather_experience(tool, probes)` | Observations (success/latency/error) | Duplicate probes deduped (diversity enforced) |79| 5 | Tool + observations | `lib.learn_from_experience(tool, obs)` | Doc-vs-reality gap report | Discovered error conditions + `gap_over_promises_any_text` |80| 6 | Tool + learning | `lib.rewrite_documentation(tool, learning)` | AI-optimized refined spec | Refined spec lists real constraints + performance, drops marketing |81| 7 | Tool + probes | `lib.run_draft(tool, probes)` | Full gather→learn→rewrite result | Refined spec's error_conditions match discovered failures |8283## Rationalizations8485| Agent rationalization | Documented rebuttal |86|------------------------|---------------------|87| "The tool description says it handles any text input — trust it." | DRAFT's worked example is exactly this: a tool claims "any text input" but fails on Unicode. Descriptions are gamed for discovery; probe the boundary before trusting the claim. |88| "Free-text capability descriptions are fine." | The chapter requires structured, enumerable capabilities with input/output types precisely because free text cannot be tested. `verify_claims` flags a capability with null input_types as unverifiable. |89| "One trust score for the tool is enough — reliability is fixed." | Trust is performance-based and evolves: successes up, failures and slow calls down. A tool that degrades (technical debt, maintenance windows) must lose trust over time, not keep a stale score. |90| "Just log the errors when the tool fails." | Learning-from-experience is systematic discovery of TRUE capabilities, not error logging. It converts observed failures into refined constraints (max length, ASCII-only) that reshape future selection. |91| "Rewriting docs is the provider's job." | Raw provider docs are written for humans and cause tool-calling errors when used as-is (Writer's finding). The rewrite — preemptive or iterative — is what makes the description an accurate agent interface. |9293## Red Flags9495- **A capability with `input_types: null` marked verifiable.** The structured-96 capability check is broken; unverifiable free-text is passing.97- **DRAFT reports `converged: true` on a tool with known undocumented limits.**98 The probe set is not diverse enough — it never hit the boundary. Add edge-case99 probes (empty, oversized, non-ASCII).100- **Trust score never moves.** `record` is not being called on real execution101 outcomes; the score is decorative.102- **Refined spec still contains marketing language.** The rewrite copied the103 claim instead of assembling from observations.104- **CLI `--help` exits non-zero.** SKILL.md / CLI mismatch; multi-harness invariant broken.105106## Non-Negotiable Verification1071081. **Run the benchmark battery.**109 ```110 python cli.py benchmark111 ```112 Confirm marketing phrases are flagged, capabilities are NOT fully verifiable113 (the sample's second capability has null input_types), the three undocumented114 constraints are discovered, and a flaky-tool history drives trust below 0.5.1151162. **Prove the gap is discovered, not assumed.**117 ```118 python cli.py learn119 ```120 The claim says "any text input"; the learning report must show121 `gap_over_promises_any_text: True` with the discovered error conditions.1221233. **Prove the refined spec drops marketing.**124 ```125 python cli.py rewrite126 ```127 The refined description states real constraints and performance; it contains128 none of the flagged marketing phrases.1291304. **JSON round-trips.**131 ```132 python cli.py draft --json | python -c "import json,sys; json.load(sys.stdin)"133 ```134135## Security Posture136137- **Prompt injection.** A gamed description can embed instructions. This skill138 never trusts the description for behavior — it probes the tool. The139 `# TODO(production):` seam in `TrustScore.record` marks where SLO-weighted140 penalties and observation decay go; the explorer's probe issuance is the141 production seam where real (sandboxed) calls replace `_simulate`.142- **Data exfiltration.** No network calls in `lib.py`; behavior is simulated143 from declarative rules. In production, run the explorer's probes in a144 sandbox so boundary-probing cannot trigger real side effects.145- **Privilege escalation.** DRAFT probes should be read-only / idempotent —146 boundary exploration must not invoke destructive tool operations. Gate probe147 generation behind the same sensitive-action policy as `information-flow-control-gate`.148149## Composition150151- **Feeds** `skill-quality-evaluator` and retrieval: the DRAFT-refined152 description improves `search` accuracy, and the trust score is a ranking153 signal for functionally-equivalent tools.154- **Feeds** `hierarchical-orchestration-router` functional clustering — DRAFT-155 refined representations are what the clusterer embeds ("embed tools based on156 what they do, not what they claim").157- **Pairs with** `information-flow-control-gate` — DRAFT learns what a tool does;158 IFC governs what data may flow into it.159- **Parallels** the MCP Gateway's preemptive description rewrite (Palmyra X5);160 DRAFT is the iterative, post-observation form of the same insight.161162## Source Attribution163164Distilled from *Agentic GraphRAG* (O'Reilly), Chapter 6 — Tool Orchestration,165sections "The Trust Problem: When Tools Game the System", "Verification-based166trust", and "DRAFT: Learning what tools actually do". Named references:167168- Baidu AI Search Paradigm — DRAFT (Documentation Refinement through Automated169 Feedback and Testing); three-phase loop170- Writer enterprise MCP gateway — preemptive description rewrite (Palmyra X5),171 the same insight applied before deployment rather than after
AnthonyAlcaraz