Improving MCP tools
An MCP server gets better only in ways you can measure. This skill is the
campaign procedure: score the current agent experience, fix the biggest
problem, re-score, and only ship changes the numbers justify. It is the
operating manual for the "improve my MCP" loop — one iteration per pass,
journaled so a later iteration (or a different agent) can resume without
repeating work.
The objective function
services/mcp/evals/ is the harness. benchmark/tasks.yaml is a fixed set of
agent tasks with expected_tools and success_criteria; scores are only
comparable across runs of the same benchmark version.
- Probe mode (deterministic, no LLM):
LIVE_MCP_URL=... LIVE_MCP_TOKEN=... pnpm exec tsx evals/runner/probe.ts --out score.json
from services/mcp/. Reports tool-presence misses (discoverability), probe
failures, and latency p50/p95. Non-zero exit = regression.
- Agent mode (LLM replay + judge): scores task success and tool-selection
accuracy. Use it for description/discoverability changes — probes cannot
detect that an agent picks the wrong tool.
Run the harness against a seeded local or devbox stack, never against a
customer project. Local recipe: NODE_ENV=development PORT=9876 POSTHOG_API_BASE_URL=http://localhost:8000 pnpm dev:hono, personal API key as
LIVE_MCP_TOKEN.
One iteration
- Measure. Run the harness for a baseline. Pull production evidence with
the MCP analytics tools (
query-mcp-tool-stats, query-mcp-tool-failures,
query-mcp-tool-descriptions, query-mcp-tool-sample-intents) and the
lenses in the signals scout cookbook
(products/signals/skills/signals-scout-mcp-tool-calls/references/queries.md):
failure leaderboard, retry/struggle, latency, intents that matched no tool.
- Pick one issue. Rank by reach × severity. Skip anything the journal
shows with two failed attempts. One issue per iteration — a PR that fixes
three things can't be attributed to any of them when scores move.
- Fix, bounded. Only files inside the allowlist (below). Typical fixes:
sharpen a tool description so the right intent finds it, tighten an input
schema that agents keep getting wrong, fix an annotation, update a skill.
- Validate. Re-run the affected benchmark slice plus a no-regression
sample. Keep the change only if the target metric improves and nothing else
degrades. A discarded change is a normal outcome — journal it and move on.
- Ship. One PR per iteration with before/after scores in the body (format
in references/campaign-journal.md). Keep
it stampable: ≤400 changed lines, only files inside the allowlist below,
apply the
stamphog label. Autonomy level comes from the campaign config —
default is draft PR for human review; only arm auto-merge when the
operator has explicitly enabled the self-driving experiment (see
guardrails).
- Journal. Append the iteration record before ending the pass.
Hard guardrails
These are not suggestions; violating any of them ends the campaign pass.
- Allowlist — a campaign PR may only touch:
products/*/mcp/tools.yaml,
products/*/skills/**, services/mcp/evals/**, the codegen outputs of
pnpm generate-tools / scaffold-yaml (services/mcp/src/tools/generated/**
and services/mcp/schema/generated-tool-definitions.json), and docs.
Anything else (handler code, package manifests, workflows, migrations, auth
paths) → stop and hand the finding to a human as a draft PR or report
instead.
- Read-only against data. The harness and all production queries are
read-only. Never create, mutate, or delete customer-visible objects while
measuring.
- Evidence or it didn't happen. No PR without a baseline score, an after
score, and the exact harness commands used.
- Benchmark integrity. Never edit
benchmark/tasks.yaml in the same PR as
a fix it validates — changing the exam and the answer together proves
nothing. Benchmark changes are their own PR and bump version.
- Budgets. Respect the operator's iteration/token/PR caps (default: stop
after 3 open unmerged campaign PRs). Two failed attempts on an issue parks
it permanently.
- Kill switch. If the campaign config, its feature flag, or the operator
says stop — stop mid-iteration, journal state, end cleanly.
Failure modes to expect
- A description change that helps one intent can steal traffic from the right
tool for another — that's why the no-regression sample is mandatory.
- Probe latency varies with stack warmth; compare medians across ≥3 runs
before attributing a latency change to your fix.
- Tool-presence misses can be feature-flag gating, not catalog absence —
check
getToolsForFeatures gating before "fixing" discoverability.
1---2name: improving-mcp-tools3description: >4 Run an improve-my-MCP campaign: an autoresearch-style loop that measures the5 MCP agent experience with the eval harness, picks the highest-impact tool6 problem from production data, makes one bounded fix, and keeps it only if7 before/after scores improve. Use when asked to "improve my MCP", run an MCP8 improvement campaign, fix tool discoverability or descriptions based on9 evidence, or prepare an eval-backed PR for a tool change. Every shipped10 change must carry eval evidence; guardrails below are hard rules.11metadata:12 source: https://skillmd.com/skills/agentcorp-inc/improving-mcp-tools13---1415# Improving MCP tools1617An MCP server gets better only in ways you can measure. This skill is the18campaign procedure: score the current agent experience, fix the biggest19problem, re-score, and only ship changes the numbers justify. It is the20operating manual for the "improve my MCP" loop — one iteration per pass,21journaled so a later iteration (or a different agent) can resume without22repeating work.2324## The objective function2526`services/mcp/evals/` is the harness. `benchmark/tasks.yaml` is a fixed set of27agent tasks with `expected_tools` and `success_criteria`; scores are only28comparable across runs of the same benchmark `version`.2930- **Probe mode** (deterministic, no LLM):31 `LIVE_MCP_URL=... LIVE_MCP_TOKEN=... pnpm exec tsx evals/runner/probe.ts --out score.json`32 from `services/mcp/`. Reports tool-presence misses (discoverability), probe33 failures, and latency p50/p95. Non-zero exit = regression.34- **Agent mode** (LLM replay + judge): scores task success and tool-selection35 accuracy. Use it for description/discoverability changes — probes cannot36 detect that an agent picks the wrong tool.3738Run the harness against a **seeded local or devbox stack**, never against a39customer project. Local recipe: `NODE_ENV=development PORT=987640POSTHOG_API_BASE_URL=http://localhost:8000 pnpm dev:hono`, personal API key as41`LIVE_MCP_TOKEN`.4243## One iteration44451. **Measure.** Run the harness for a baseline. Pull production evidence with46 the MCP analytics tools (`query-mcp-tool-stats`, `query-mcp-tool-failures`,47 `query-mcp-tool-descriptions`, `query-mcp-tool-sample-intents`) and the48 lenses in the signals scout cookbook49 (`products/signals/skills/signals-scout-mcp-tool-calls/references/queries.md`):50 failure leaderboard, retry/struggle, latency, intents that matched no tool.512. **Pick one issue.** Rank by reach × severity. Skip anything the journal52 shows with two failed attempts. One issue per iteration — a PR that fixes53 three things can't be attributed to any of them when scores move.543. **Fix, bounded.** Only files inside the allowlist (below). Typical fixes:55 sharpen a tool description so the right intent finds it, tighten an input56 schema that agents keep getting wrong, fix an annotation, update a skill.574. **Validate.** Re-run the affected benchmark slice plus a no-regression58 sample. Keep the change only if the target metric improves and nothing else59 degrades. A discarded change is a normal outcome — journal it and move on.605. **Ship.** One PR per iteration with before/after scores in the body (format61 in [references/campaign-journal.md](references/campaign-journal.md)). Keep62 it stampable: ≤400 changed lines, only files inside the allowlist below,63 apply the `stamphog` label. Autonomy level comes from the campaign config —64 default is **draft PR for human review**; only arm auto-merge when the65 operator has explicitly enabled the self-driving experiment (see66 guardrails).676. **Journal.** Append the iteration record before ending the pass.6869## Hard guardrails7071These are not suggestions; violating any of them ends the campaign pass.7273- **Allowlist** — a campaign PR may only touch: `products/*/mcp/tools.yaml`,74 `products/*/skills/**`, `services/mcp/evals/**`, the codegen outputs of75 `pnpm generate-tools` / `scaffold-yaml` (`services/mcp/src/tools/generated/**`76 and `services/mcp/schema/generated-tool-definitions.json`), and docs.77 Anything else (handler code, package manifests, workflows, migrations, auth78 paths) → stop and hand the finding to a human as a draft PR or report79 instead.80- **Read-only against data.** The harness and all production queries are81 read-only. Never create, mutate, or delete customer-visible objects while82 measuring.83- **Evidence or it didn't happen.** No PR without a baseline score, an after84 score, and the exact harness commands used.85- **Benchmark integrity.** Never edit `benchmark/tasks.yaml` in the same PR as86 a fix it validates — changing the exam and the answer together proves87 nothing. Benchmark changes are their own PR and bump `version`.88- **Budgets.** Respect the operator's iteration/token/PR caps (default: stop89 after 3 open unmerged campaign PRs). Two failed attempts on an issue parks90 it permanently.91- **Kill switch.** If the campaign config, its feature flag, or the operator92 says stop — stop mid-iteration, journal state, end cleanly.9394## Failure modes to expect9596- A description change that helps one intent can steal traffic from the right97 tool for another — that's why the no-regression sample is mandatory.98- Probe latency varies with stack warmth; compare medians across ≥3 runs99 before attributing a latency change to your fix.100- Tool-presence misses can be feature-flag gating, not catalog absence —101 check `getToolsForFeatures` gating before "fixing" discoverability.