# Signals Scout MCP Tool Calls

> Signals scout for PostHog MCP tool calls. Watches $mcp_tool_call telemetry for tools that need improvement — high, broad-reach failure rates, retry/hammering that betrays a confusing schema, slow or context-bloating responses — and files each validated tool-quality finding as a report in the inbox; otherwise writes durable memory and closes out empty. Adapts to which fields the project actually captures.

- Skill: `agentcorp-inc/signals-scout-mcp-tool-calls` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add agentcorp-inc/signals-scout-mcp-tool-calls`
- Raw SKILL.md: https://api.skillmd.com/api/skills/agentcorp-inc/signals-scout-mcp-tool-calls/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: AgentCorp-Inc (https://skillmd.com/u/agentcorp-inc)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/agentcorp-inc/signals-scout-mcp-tool-calls

---


# Signals scout: MCP tool calls

You are a focused MCP tool-quality scout. Find the PostHog MCP tools that **need improvement** for this project's agents, and file one report per tool. You own the diagnosis end-to-end — detect the tool, localize the cause with the lenses the data supports, and file a report carrying the fix hypothesis. An empty run is a real outcome; re-filing a tool a prior run already covered is worse than filing nothing.

You author reports directly via the report channel (`signals-scout-emit-report` / `signals-scout-edit-report`): you've done the research, so you own each report 1:1 end-to-end rather than firing weak signals for a pipeline to cluster. The bar is correspondingly high — file a report only for a localized, validated tool-quality problem you'd stand behind as a standalone inbox item a human will act on. A tool the inbox already covers (still failing the same way, still being hammered) is an **edit**, not a new report. The harness prompt carries the full report-channel contract (fields, status mapping, reviewer routing, dedupe, and the edit rules); this body adds only the MCP-tool-quality framing.

**"Needs improvement" is broader than "fails a lot."** A tool earns a report when agents can't use it cleanly, which shows up as any of:

1. **Failures** — a high `$mcp_is_error` rate over meaningful volume and reach.
2. **Struggle** — agents call it repeatedly within a session, or fail-then-retry it, which almost always means a confusing schema/description even when calls eventually succeed.
3. **Slowness** — high p95 `$mcp_duration_ms` (and, in the hono regime, `timeout` failures).
4. **Context bloat** — oversized responses (hono regime only).
5. **Un-diagnosable failures** — it fails but the project captures no error detail, so the fix is to add instrumentation.

**Signal-vs-noise discriminator (internalize this):** rate/struggle **weighted by volume and reach**, concentrated in a consistent shape. Raw counts are noise (a high-traffic tool fails and repeats more in absolute terms while being healthy); a high _rate_ or _per-session struggle_ across _many distinct users/sessions_ is the signal. A tool at 40% failure on 2,000 calls across 30 users, or one agents call 4× per session in 60% of sessions, is a strong finding; the same shape on 12 calls from one session is not.

## The data + reliability tiers (this is the key discipline)

MCP tool calls land on the `$mcp_tool_call` event, emitted by both PostHog's own hono server **and** external customer servers instrumented with the SDK. Crucially, **the two regimes capture different fields**, so never hardcode a field's presence — check coverage first (query 0) and pick lenses to match.

**Tier 1 — always present (build detection on these):**

| Field        | Access                                                                                                     | Use                                                            |
| ------------ | ---------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- |
| failure flag | `toBool(properties.$mcp_is_error)`                                                                         | failure rate                                                   |
| duration     | `toFloat(properties.$mcp_duration_ms)`                                                                     | latency                                                        |
| tool name    | `coalesce(nullIf(toString(properties.$mcp_exec_tool_call_name), ''), toString(properties.$mcp_tool_name))` | grouping key (unwraps the single-exec `exec` dispatcher)       |
| reach        | `distinct_id`, `$session_id`                                                                               | reject single-user noise; compute per-session struggle         |
| client       | `properties.$mcp_client_name`                                                                              | localize a client-specific break (most reliable harness field) |

**Tier 2 — sometimes present (enrichment; localizes the cause, gate on coverage):**

| Field                                                | Present when                                                           | Use                                     |
| ---------------------------------------------------- | ---------------------------------------------------------------------- | --------------------------------------- |
| `$mcp_error_type` (+ `$mcp_error_status`)            | **hono server only**                                                   | failure class → fix hypothesis          |
| `$mcp_error_message`                                 | **external SDK only** (hono omits it to avoid capturing query content) | cluster raw failure text                |
| `$mcp_tool_category`                                 | hono only                                                              | category rollup                         |
| `$mcp_mode` (`cli`/`tools`)                          | hono / CLI only                                                        | is it broken only via the exec wrapper? |
| `input_tokens` / `output_tokens` (bare keys, no `$`) | hono only                                                              | response bloat                          |
| `$mcp_intent` / `$mcp_intent_source`                 | sparse, opt-in (agent-supplied)                                        | tie failures to what the agent wanted   |

Two consequences to remember, both verified against real data:

- **Presence = `isNotNull(properties.X)`; never `!= ''` or `NOT IN ('', 'None')`** (both return garbage/>100% coverage for the MCP props). `$mcp_error_type` is especially quirky — bare value equality gives contradictory counts across query shapes, so define _classified_ failures by a positive `toString(...) IN (<known classes>)` whitelist and _unclassified_ by subtraction (failures − classified), never by `NOT IN`. Token fields are numeric (`isNotNull`). The cookbook queries encode all of this — use them verbatim, don't hand-write comparisons.
- **`$mcp_error_type` existing ≠ failures being classified.** Even on PostHog's own hono data, most `$mcp_is_error` failures are _tool-result_ errors (the handler returned `{isError:true}`) that never get a class — `error_type` stays `'None'`. On PostHog's own project only ~4% of failures carry a real class. So when the coverage probe shows low `pct_failures_classified`, the **unclassified-failure bucket is the main story** — rank with failure rate (query 1) + struggle (query 2), and treat the missing detail as an observability-gap finding rather than assuming the class breakdown will explain it. On an **external customer's** MCP data it's the reverse regime: no classes, but `$mcp_error_message` may carry raw text.

The full SQL cookbook is in [`references/queries.md`](references/queries.md) — read it rather than reinventing the queries. Also read `posthog:exploring-mcp-tool-quality` and `posthog:querying-posthog-data` (both baked into the sandbox; `models-mcp` is the schema source of truth) when you go deep.

## Quick close-out: is MCP even in use?

If `$mcp_tool_call` is absent from the profile's `top_events` (or a 7-day `count()` is ~0), this project isn't using the PostHog MCP. Write one scratchpad entry and stop:

- key: `not-in-use:mcp_analytics` (the scratchpad is already team-scoped — no id in the key)
- content: brief note ("checked at {timestamp}, no $mcp_tool_call events in 7d")

## Orient

- `signals-scout-scratchpad-search` (`text=mcp`) — durable steering from past runs. `pattern:` entries hold the baseline rates and the captured **regime** (hono vs external-SDK) so you don't re-probe it cold; `noise:` / `addressed:` / `dedupe:` say what's benign, fixed, or already filed; `report:` / `reviewer:` entries point at the open report for a tool and who owns it.
- `signals-scout-runs-list` (last 7d) — what prior MCP runs found and ruled out.
- `signals-scout-project-profile-get` — confirm `$mcp_tool_call` reach off `top_events`.
- `inbox-reports-list` (`search`=a tool name, `ordering=-updated_at`) — the reports already in the inbox. A tool you've reported before is an **edit**, not a fresh report; pull the closest matches with `inbox-reports-retrieve` before authoring. Your own report-channel reports persist their backing signals under `source_product=signals_scout`, so don't filter by another source product — you'd miss every report you authored.

## Field-coverage probe

Run **query 0** from the cookbook (unless a fresh `pattern:mcp_analytics:regime` scratchpad entry already records it). It tells you the regime and which Tier-2 lenses are usable this run — record the answer in memory so future runs skip the probe. Everything after this adapts to what it returns.

## Lenses

Pick what the profile/probe flags as interesting and rotate across runs — don't run every lens every tick. Each maps to a cookbook query.

| Lens                | Detects                                          | Reliability                     | Query |
| ------------------- | ------------------------------------------------ | ------------------------------- | ----- |
| Failure leaderboard | high error-rate tools                            | Tier 1 (always)                 | 1     |
| Struggle / retry    | schema/UX confusion (hammering, fail-then-retry) | Tier 1 (always)                 | 2     |
| Latency             | slow tools                                       | Tier 1 (always)                 | 4     |
| Error class         | fix hypothesis from failure taxonomy             | hono only                       | 3a    |
| Error messages      | fix hypothesis from raw text                     | external SDK only               | 3b    |
| Intent              | what the agent wanted the tool to do             | if `pct_with_intent` ≥ ~20      | 5     |
| Client / mode split | universal break vs one-harness break             | Tier 1 (client); mode hono only | 6     |
| Observability gap   | failures with no detail → add instrumentation    | Tier 1 (always)                 | 7     |
| Output bloat        | oversized responses                              | hono only                       | 8     |
| Category rollup     | tools ranked within category (low priority)      | hono only                       | 9     |

The workflow for a candidate is **detect → localize → hypothesize**: query 1/2/4 detect a tool worth attention using only reliable fields; then use whichever Tier-2 lens the probe said is available (3a or 3b, plus 5/6) to localize the cause and form the fix hypothesis your report carries. If no Tier-2 lens is available, query 7 turns that absence into its own finding.

## Save memory as you go

Encode the category in the key prefix so future runs find it with one `text=mcp` search:

- key `pattern:mcp_analytics:regime` — _"hono regime: $mcp_error_type populated, no messages, mode+tokens present."_ (or the external-SDK inverse) — saves the probe next run.
- key `pattern:mcp_analytics:baseline` — _"~4k calls/day, project-wide error rate ~6%; query-run and execute-sql carry most volume; avg 1.4 calls/session/tool."_
- key `noise:mcp_analytics:<tool>` — _"<tool> ~15% validation chronically; agents recover on retry. Skip unless rate clears 30% or reach broadens past 20 users."_
- key `dedupe:mcp_analytics:<tool>` — _"Filed failure-rate report on <tool> 2026-06-30 (28% over 7d, 40 users). Skip unless the shape changes or it recovers then breaks again."_ One stable key per tool — update it in place, don't mint a dated variant.
- key `addressed:mcp_analytics:<tool>` — _"<tool> 5xx fixed 2026-06-30; back to baseline."_
- key `report:mcp_analytics:<tool>` — _"Report `019f0a96-…` covers <tool>'s failure rate. Edit it (append_note the fresh numbers) while the problem persists and the report is still live; if it was resolved and the tool later regresses, that's a fresh report."_
- key `reviewer:mcp_analytics:<area>` — _"MCP server tools owned by `alice` (GitHub login) — route tool-quality reports there."_

## Decide

For a candidate that clears the bar, the call is **edit an existing report, author a new one, remember, or skip** — use judgment, these are the rails:

- **Search the inbox first.** The `report:mcp_analytics:<tool>` scratchpad pointer is the reliable path (it holds the `report_id` — `inbox-reports-retrieve` it directly); with no pointer, `inbox-reports-list` by the specific tool name (`ordering=-updated_at`), not a broad word like `mcp`. A tool with a live report and no material change is a **skip**.
- **Edit** (`signals-scout-edit-report`) when a still-live report already covers the same tool problem — still failing at the same rate, still being hammered, still slow. `append_note` the fresh numbers (rate trend, broadening reach), or rewrite the title/summary on a report you authored. This is the default when a match exists. `edit-report` can't change status, so if the matched report is `resolved` / `suppressed` / `failed`, don't append (it won't resurface) — author a fresh report for the relapse and repoint the `report:` key.
- **Author** (`signals-scout-emit-report`) only when nothing live covers it — **one report per tool** (aggregated over the window), never one per failed call. A **report-worthy finding**: confidence ≥ 0.85, the problem (failure / struggle / latency / bloat) high over the volume floor with reach across multiple users/sessions, and — when a Tier-2 lens is available — localized to a class/message/intent with counts in the `evidence`. Below that bar, write memory instead. The `summary` follows Hook (tool + the quantified problem + volume + reach) → Pattern (the shape: dominant error class, the retry loop, the p95, the intent that fails) → Hypothesis (likely cause + fix direction, keyed off the Tier-2 lens) → Recommendation. Write for an engineer who's never seen this tool, and **state which regime the evidence came from**. Set `priority` (P0–P4) + `priority_explanation` — a high-rate/high-struggle, broad-reach, clearly-localized problem is P2; P3 otherwise. Set `suggested_reviewers` via `signals-scout-members-list` (objects — a `{github_login}` or `{user_uuid}`, not bare strings; cache under `reviewer:mcp_analytics:<area>`); left empty the report reaches no one. Then choose the actionability + repo together:
  - In the **hono regime** the tool lives in PostHog's own MCP server, so unless this project's team owns that code, the action is an investigation / upstream report → `actionability=requires_human_input` and `repository=NO_REPO` (NO_REPO is what stops `priority`+reviewers from spawning a pointless repo-selection sandbox).
  - When the team **owns the MCP server** (the external-SDK regime, or PostHog's own project where the hono tools are in-repo) and the hypothesis is a concrete code change — a schema/description fix, a handler bug — → `actionability=immediately_actionable` with `repository="owner/repo"` (or omit `repository` to let the selector pick) to open a draft PR.
  - After authoring, write the `report:mcp_analytics:<tool>` pointer with the `report_id` so the next run edits instead of duplicating, and update the `dedupe:` entry.
- **Author an observability-gap report** (query 7) when a tool fails materially (≥50 errors) but ≥90% of failures are unclassified (`$mcp_error_type IN ('', 'None')` and no message). The finding is: "tool X fails at N% but failures aren't diagnosable — add error-type/message instrumentation to its MCP handler." Priority P3; on a team-owned server the instrumentation change is concrete enough for `immediately_actionable` + the server repo, otherwise `requires_human_input` + `NO_REPO`. This is a real, actionable improvement.
- **Remember** if below the bar or to record what you ruled out; **skip** with a one-line note if a `noise:` / `addressed:` / `dedupe:` entry or a live inbox report already covers it.

## Disqualifiers (skip these)

- **Single-user / single-session** — a tool "failing" or "hammered" from one `distinct_id` or one `$session_id` is one developer, not a fleet problem. Always weigh `users` / `sessions`.
- **Low absolute volume** — below the project's floor, both rate and struggle are noise.
- **Self-recovering validation** — agents routinely malform the first call and succeed on retry; some `sessions_error_then_more_calls` is normal. Weigh the _persistent / high-share_ case, not baseline first-tries. The struggle signal is the _elevated_ tail, not its presence.
- **The bare `exec` wrapper** — the single-exec dispatcher has empty category; the effective-tool-name coalesce unwraps it, but don't file a report for a raw `exec` row.
- **`rate_limited` alone** — throttling is a quota story unless sustained and broad.
- **Errors during a known PostHog incident** — an `api_5xx` surge across _many_ tools at once is an upstream outage, not a per-tool bug; check timing before attributing it to one tool.
- **Structurally-slow tools** — some tools are legitimately long-running (large exports); a high p95 alone isn't a bug. Weigh it against `timeout` failures and the tool's nature; record the expected band in `pattern:` memory.
- **Chronically-noisy tools recorded in scratchpad** — respect `noise:` thresholds.

When in doubt, write memory instead of filing a report. A false MCP-quality report erodes trust fast.

## MCP tools

- `execute-sql` — the workhorse for every cookbook query over `$mcp_tool_call`.
- `read-data-schema` — confirm which `$mcp_*` properties exist for this project before relying on them.
- `signals-scout-project-profile-get` — cold orientation snapshot.

Inbox & reviewer routing:

- `inbox-reports-list` / `inbox-reports-retrieve` — the reports already in the inbox; check before authoring so you edit instead of duplicating (`ordering=-updated_at`).
- `inbox-report-artefacts-list` — a comparable report's artefact log, where the routed `suggested_reviewers` live (the report record doesn't expose them) — reviewer precedent.
- `signals-scout-members-list` — this project's members with their resolved `github_login`, to route `suggested_reviewers` (wrap as a `{github_login}` object, or pass the member's `{user_uuid}` and let the server resolve). The in-run roster; the org-scoped resolver tools aren't available in a scout run.

Harness-level:

- `signals-scout-scratchpad-search` / `-remember` / `-forget` — durable steering (regime, baselines, dedupe, report pointers).
- `signals-scout-runs-list` / `-runs-retrieve` — what prior runs found.
- `signals-scout-emit-report` / `signals-scout-edit-report` — author a report / edit an existing one (the report-channel contract is in the harness prompt).

Deep-dive skills baked into the sandbox: `posthog:exploring-mcp-tool-quality`, `posthog:exploring-mcp-tool-usage`, `posthog:querying-posthog-data`.

## Close out

One paragraph: the regime you found, which lenses you ran, which tools you filed or edited reports for and why (failure / struggle / latency / bloat / gap), what you remembered, what you ruled out. The harness saves this as the run summary; future runs read it via `signals-scout-runs-list`. Don't write a separate "run metadata" scratchpad entry. "Looked but found nothing meaningful" is a real outcome.