# Prod Telemetry

> Query Executor's production telemetry — Axiom traces (executor-cloud dataset), prod Postgres via PlanetScale, PostHog product analytics — through the Executor MCP. Use when investigating prod errors, latency, usage, churn signals, or verifying a deploy's telemetry; includes the dataset field layout, working APL recipes, and the error-attribution join.

- Skill: `rhyssullivan/prod-telemetry` (Agent Skill)
- Install (CLI): `npx skillmds@latest add rhyssullivan/prod-telemetry`
- Raw SKILL.md: https://api.skillmd.com/api/skills/rhyssullivan/prod-telemetry/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: rhyssullivan (https://skillmd.com/u/rhyssullivan)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/rhyssullivan/prod-telemetry

---


# Production telemetry access

All three stores are queryable through the Executor MCP's connected
integrations — no dashboards or credentials needed. Verify the connection
exists with `connections.list` if a call fails.

## Axiom traces (`axiom_mcp`)

Tool: `axiom_mcp.user.axiomMcpOAuth.querydataset` — the argument is `apl`
(NOT `query`). Dataset: `['executor-cloud']` (worker spans; browser spans
join the same traces via traceparent).

**Field layout (the part you'd otherwise rediscover by failed queries):**

- Custom span attributes live under the JSON map `['attributes.custom']`,
  NOT as top-level `attributes.*` columns. Read with
  `['attributes.custom']['mcp.tool.name']`. A nonexistent top-level field is
  a hard query error ("invalid field"), not an empty result.
- Span status: `['status.code']` (`"OK"`/`"ERROR"`), `['status.message']`.
- Exceptions: the `events` column carries `exception.type` /
  `exception.stacktrace` JSON.
- OTel basics are top-level: `name`, `trace_id`, `span_id`,
  `parent_span_id`, `duration`, `_time`.

**Span names worth querying** (and their custom attrs):

- `mcp.execute` / `mcp.execute.resume` — `mcp.execute.mode`
  (`pausable`/`inline`), `mcp.execute.code_length`, and
  `mcp.execute.outcome` (`ok`/`fail`/`paused`) with, on failures,
  `mcp.execute.error_kind` (`type_error` | `reference_error` |
  `syntax_error` | `range_error` | `tool_error` | `timeout` |
  `resource_limit` | `serialization_error` | `thrown` | `unknown`).
  Sandbox script failures ride the MCP success channel, so `status.code`
  stays OK — filter on these attributes, not span status. Spans from
  before the attributes shipped carry neither; absence is not success.
  Also `mcp.execute.result_chars` (compact-JSON size of the returned
  value, pre-truncation; -1 = unmeasurable), `mcp.execute.log_chars`,
  `mcp.execute.emitted` — the dump-vs-narrow signal (the model preview
  truncates at 30k chars, so `result_chars > 30000` means the model tried
  to pull a truncated blob into context).
- `executor.tool.execute` — `mcp.tool.name` (full address), and since
  PR #992: `executor.tool.outcome` (`ok`/`fail`),
  `executor.tool.error_code`, `executor.tool.error_status`,
  `executor.tenant`, `executor.subject`.
- `mcp.tool.dispatch` — `mcp.tool.name` (sandbox path),
  `mcp.tool.integration`, same outcome attrs.
- `plugin.openapi.invoke` — `plugin.openapi.method` / `path_template` /
  `base_url`, and since PR #992 `http.status_code`.
- `mcp.request` (outer) — `mcp.auth.organization_id`,
  `mcp.auth.account_id`, `mcp.tool.name`, CF edge fields (`cf.country`…),
  MCP client fingerprint (`mcp.client.name`…), and on managed-cloud
  `execute`/`execute-action` calls `mcp.execute.code` (the script itself,
  capped at 10k chars — cloud-only content capture; local/self-host
  telemetry never records content).

**Recipe — error signatures by class (the daily-digest query):**

```apl
['executor-cloud']
| where _time > ago(1d)
| where ['status.code'] == "ERROR" and name == "executor.tool.execute"
| extend msg = substring(tostring(['status.message']), 0, 120)
| extend tool = tostring(['attributes.custom']['mcp.tool.name'])
| summarize n = count() by msg, tool
| sort by n desc
```

**Recipe — attribute errors to orgs.** Tool spans now carry
`executor.tenant` directly (post-#992). For spans from BEFORE that deploy,
join through the outer request span:

```apl
['executor-cloud']
| where name == "mcp.request" and isnotnull(['attributes.custom']['mcp.auth.organization_id'])
| project trace_id, org = tostring(['attributes.custom']['mcp.auth.organization_id'])
| join kind=inner (
    ['executor-cloud']
    | where ['status.code'] == "ERROR" and name == "executor.tool.execute"
    | project trace_id, msg = substring(tostring(['status.message']), 0, 60)
  ) on trace_id
| summarize n = count() by org, msg | sort by n desc
```

**Recipe — upstream failure rate per integration (post-#992 attrs):**

```apl
['executor-cloud']
| where _time > ago(1d) and name == "mcp.tool.dispatch"
| extend outcome = tostring(['attributes.custom']['executor.tool.outcome'])
| extend integration = tostring(['attributes.custom']['mcp.tool.integration'])
| where isnotnull(outcome)
| summarize calls = count(), fails = countif(outcome == "fail") by integration
| extend failRate = todouble(fails) / todouble(calls)
| sort by fails desc
```

**Known signal caveats** (audited 2026-06-12):

- Pre-#992 spans: `ToolResult.fail` outcomes (upstream 4xx/5xx, auth
  rejections) are INVISIBLE — they rode the Effect success channel with no
  span marker. Don't conclude "no errors" from old data.
- Many pre-#992 ERROR spans have an EMPTY `status.message` (tagged errors
  without a message field) — group those by `events` exception.type instead.
- `[object Object]` status messages are the pre-#992 formatting bug.

## Prod database (`planetscale_mcp`)

Read tool needs `{organization: "answer-overflow", database: "executor",
branch: "main"}`. It returns `ok: true` even when the SQL failed — check the
result text for `Error:`. Use for tenant/integration/connection facts that
spans don't carry (row sizes, config shapes, counts).

## Product analytics (`posthog_api` / `mcp_posthog_com`)

Browser-side events only (the ~60-event typed catalog, PR #987; server-side
events not built). The org-key `posthog_api` connection covers the REST API;
the OAuth MCP connection covers the higher-level tools.

## Verifying a deploy's telemetry (Layer-0 canary)

After deploying telemetry changes: run a known-failing tool call against
prod, then assert the expected attributes arrive in Axiom within ~1 min.
Absence of data looks identical to health — query for the NEW attribute
explicitly rather than eyeballing dashboards. The e2e equivalent runs on
every suite: `e2e/cloud/telemetry-contract.test.ts` via the `Telemetry`
service (motel `/api/spans/search?attr.<key>=<value>`).