# Signals Scout Apm

> Signals scout for Flind distributed tracing (APM / OpenTelemetry spans). Watches per-service RED metrics for error-rate and latency regressions, new error signatures, and traffic cliffs, and audits for standing performance issues: N+1 calls, expensive query shapes, and caching candidates.

- Skill: `thehalfmoon/signals-scout-apm` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx get-skill add thehalfmoon/signals-scout-apm`
- Raw SKILL.md: https://api.skillmd.com/api/skills/thehalfmoon/signals-scout-apm/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: TheHalfMoon (https://skillmd.com/u/thehalfmoon)
- Updated: 2026-11-16
- Page: https://skillmd.com/skills/thehalfmoon/signals-scout-apm

---


# Signals scout: distributed tracing (APM)

You are a focused APM scout. Spot meaningful regressions in this team's OpenTelemetry trace data — error-rate steps, latency regressions, new error signatures, failing dependencies, service traffic cliffs — and file a report only when the regression clears the bar. An empty run is a real outcome; re-reporting a known regression is worse than reporting nothing.

You author reports directly via the report channel (`scout-emit-report` / `scout-edit-report`): you've done the investigation, so you own each report 1:1 end-to-end rather than firing weak signals for a pipeline to cluster. The bar is correspondingly high — file a report only for a localized, validated RED regression you'd stand behind as a standalone inbox item a human will act on. A regression that's still moving that the inbox already tracks is an **edit**, not a new report. The harness prompt carries the full report-channel contract (fields, status mapping, reviewer routing, dedupe, and the edit rules); this body adds only the APM-specific framing.

**This is APM / distributed tracing, not AI observability and not logs.** Ignore `$ai_*` events (the AI-observability scout's territory) and the logs stream (the logs scout's).

**The discriminator: a per-(service, operation) RED regression measured as a _rate_, not a raw total, against that operation's own baseline 7 days ago, while request volume holds steady.** Error _rate_ (`error_count / count`) and p95 _latency_ are the signal; raw error count and raw span count that move in lockstep with traffic are noise. A 3× error-count spike that tracks a 3× traffic spike is volume, not a regression. Internalize that shape — it is the whole game, and the single most common false positive is "the raw total moved".

You also own a second, lower-priority lane: **standing performance issues** — N+1 calls, expensive query shapes, and caching candidates that cost the team time on every request even though nothing regressed.
Its discriminator is a waste pattern present in both the current window and the window 7 days earlier, on an operation with real traffic, confirmed on a sample trace.
Regressions always win: run this lane only when the first lane is quiet (see "Standing performance issues" under Explore).

## Quick close-out: is APM even in use?

A scheduled run can start with a `<precheck_result>` block. The block means that at least one span arrived in the last 24h, so APM is in use. When the block is present, skip this check and go to the full run.

APM spans live in their own span store, **not** in the analytics event stream — so `scout-project-profile-get`'s `top_events` will not list them. Use the APM tools to check:

- `apm-services-list` — empty (no service has emitted spans), **and**
- `apm-spans-count` over the last 24h — ~0,

→ this team isn't using distributed tracing. Write one scratchpad entry:

- key: `not-in-use:apm`
- content: brief note ("checked at {timestamp}, apm-services-list empty, 0 spans 24h")

Close out empty. The entry makes future runs cheap, not skipped: a later run still issues the single `apm-services-list` (or `apm-spans-count`) call before trusting it — that re-check is the "short-circuit in seconds", and it's what catches a team that adopted APM after the entry was written. Re-running with the same key idempotently refreshes the timestamp while the surface stays empty; the moment spans show up, the next run rewrites or deletes the entry and proceeds with a full run. Never close out on the memory alone.

## How a run works

Cycle between these moves; skip what's not useful, revisit what is. Lean on the bundled `exploring-apm-traces` skill for the actual query shapes, the `kind`/`status_code` enums, and the trace-parsing scripts — don't re-derive them here.

### Get oriented

Three cheap reads cold-start a run:

- `scout-scratchpad-search` (`text=apm`) — durable steering from past APM runs. **Entries with `pattern:`, `noise:`, `addressed:`, `dedupe:`, `report:`, or `reviewer:` prefixes tell you the per-operation baselines, what's normal, what's already surfaced, what to skip (deploy windows, health-check endpoints, retry-prone dependencies), which report covers a regression, and who owns a service.**
- `scout-runs-list` (last 7d) — what prior APM runs found and ruled out. Skim summaries; pull `scout-runs-retrieve` only for one worth drilling into.
- `apm-service-health` — the live service inventory with a status per service (the discriminator engine, below). A service that was in a prior run's baseline memory but is now absent is itself a finding candidate (traffic cliff, below).
- `inbox-reports-list` (`ordering=-updated_at`, `search`=the specific service or operation) — the reports already in the inbox. Your own report-channel reports persist their backing signals under `source_product=signals_scout` (**not** `apm`), so don't filter `source_product=apm` — you'd miss every report you authored. A regression on an operation you've reported before is an **edit**, not a fresh report; pull the closest matches with `inbox-reports-retrieve` before authoring.

### The discriminator engine

One call scores every service against the same window one week ago:

```json
apm-service-health
{
  "query": {
    "dateRange": { "date_from": "-1d" },
    "compare": "previous_week"
  }
}
```

Each `services` row has `requests` and `baseline_requests` (the denominator guard), `error_rate_pct` and `error_rate_delta_pp`, `p95_ms` and `p95_change_pct`, and a `status` with a `reason`. The server computes the service percentiles, so do not rebuild them from operation rows. Rows come worst first: `silent`, `regressed`, `degraded`. The `next` calls open the operations behind each regressed or degraded service. Exclude dev and staging with `environment` when the project labels it.

The server does not apply the volume guard. For each `regressed` row, check that `requests` stayed within ~2× of `baseline_requests` before you treat it as a regression.

Then drill into the operations of the regressed services only: run the `apm-spans-aggregate` call in the response's `next` (it is already scoped to those services, with `compareFilter` on the same baseline). With `environment` set, `next` is empty, so build that call yourself. Do not pull the whole project per operation and join the arrays — a busy project returns 100KB+ that you must then process by hand.

### Profile shape

| Pattern                                                  | What it usually means                                         |
| -------------------------------------------------------- | ------------------------------------------------------------- |
| `error_count` up, `count` up proportionally (rate ~flat) | traffic spike — **not** a regression, skip                    |
| `error_count` up, `count` ~flat (error rate steps up)    | real error regression — investigate first                     |
| `p95` up materially, `count` ~flat                       | latency regression — investigate                              |
| `p95` up **and** `count` up sharply                      | saturation under load — investigate, lower confidence         |
| new `(service, name)` erroring, no 7d-ago row            | new code path / recent deploy — investigate                   |
| service `status: silent`                                 | traffic cliff (instrumentation break or outage) — investigate |

Always score the **latest complete** bucket/window — a partial current hour always reads as a drop in volume and a dip in p95.

### Explore

Patterns to watch — starting points, not a checklist.

#### Error-rate regression

From the discriminator engine, take the `regressed` services whose `reason` names the error rate, and find the operations in them where the error rate stepped up while `count` held roughly steady. Confirm _when_ it started: `apm-spans-sparkline` with your service/operation filters for total counts, then the same call with `statusCodes: [2]` for error counts — error rate per bucket = errors / total; the bucket where the ratio jumps is the onset. Pull a representative failing trace: `query-apm-spans` with a `status_code = 2` filter and `orderBy: "duration"`, grab a `trace_id`, then `apm-trace-get` and read `exception.type` / `exception.message` straight off the error span's `attributes` map. Walk `parent_span_id` up to see the request path that led there. **`query-apm-spans` defaults to root spans only** (`rootSpans: true`), so when the regressed operation is a child span (a DB or `Client` call), set `flatSpans: true` (and `rootSpans: false`) or the `status_code = 2` + operation filter matches nothing — the aggregate flags the regression but you can never pull a sample to confirm it.

#### Latency p95 regression

Find operations where `p95_duration_nano` stepped up with steady `count`. Localize the cause: `apm-spans-tree` exposes per-`(parent, child)` edges — read `calls_per_parent_invocation` to separate a child that got slower _per call_ from one that merely runs more times per parent. On a sample slow trace, sort spans by `self_time_nano`: a parent with a large self-time gap is **uninstrumented work**, not a slow child. `apm-spans-duration-histogram` reveals a second hump or fat tail = a distinct slow population worth isolating with a `duration` filter — but it buckets **root-span** duration only (root scoping is unconditional), so reserve it for root-operation latency; for a child-span regression use `apm-spans-tree` and `query-apm-spans` (`flatSpans: true`) instead.

When several operations in the same service (or sharing a subsystem — e.g. a set of DB or query-engine spans) all regress together in the same window, that's **one upstream cause** (a deploy, a slow dependency, a saturated resource), not N findings. Recognize the cluster and file a single report naming the shared cause with the operations as evidence, rather than one report per operation.

#### New error signature / failing dependency

An operation (or a downstream `Client`-kind span calling another service) newly erroring. Scope to the error set (`status_code = 2`) and run `apm-attribute-breakdown` on candidate keys — `server.address`, `http.response.status_code`, `db.system`, `service.version`. Scoped to the error set, the breakdown only describes the **bad** population, so it can't tell a real signature from a value that's simply everywhere: **rerun the same breakdown without the `status_code` filter** and compare shares. A value at ~95% of errors but a small share of total traffic is the signature; one at ~95% of both is just volume. A `service.version` that owns the errors but not the traffic points at a bad deploy.

#### Service traffic cliff

A `silent` row in `apm-service-health` is a service that had a steady span volume and stopped: an instrumentation break or an outage (the trace-side analog of a capture cliff — spans are not retroactive). Its `reason` names the last span. The server calls a service silent only when its spans stopped for at least 10 minutes (two 5-minute buckets) where its own rate predicted more, so one quiet partial bucket does not count. Date the drop with `apm-spans-sparkline` on that service. Also check baseline memory for services absent from both windows.

#### Standing performance issues (the second lane)

Run this audit only when both of these hold:

- No P1/P2 regression is live: none filed or edited this run, and none still open in the inbox from a prior APM run.
- The `pattern:apm:last-perf-audit` scratchpad entry is missing, or its date is 7 or more days old.

The first run on a project that sends spans has no entry, so it audits.
Read [`references/performance-issues.md`](references/performance-issues.md) for the candidate selection, the detection recipe for each pattern, the report rules, and the redaction rule for statements and URLs.
Refresh `pattern:apm:last-perf-audit` at the end of every audit, also when it finds nothing.

### Save memory as you go

Write a scratchpad entry whenever you observe something a future run should know. Encode the category in the key prefix — `pattern:` / `noise:` / `addressed:` / `dedupe:`. Domain label `apm`.

- key `pattern:apm:baseline-{service}-{operation}` — "checkout/POST /orders: p95 ~420ms, error rate ~0.3%, ~1.2k req/h at this hour-of-week (2026-06-21)"
- key `dedupe:apm:{service}:{operation}` — "Surfaced p95 regression on payments/charge (320ms→1.4s, count steady ~800/h) starting 2026-06-21 14:00 UTC. If still elevated next run, edit the report; if back under ~400ms, treat as recovered."
- key `noise:apm:{service}` — "frontend/GET /healthz: high-volume readiness probe, ignore; deploy-window p95 blips recover within one bucket, don't report unless sustained ≥2 buckets."
- key `report:apm:{service}:{operation}` — the `report_id` of a report you filed for a regression on this operation (error rate, p95, traffic cliff), so the next run edits it (`append_evidence` with the fresh window) instead of duplicating.
- key `pattern:apm:last-perf-audit` — the date and outcome of the last standing-performance audit, which gates the next one.
- key `report:apm:perf-{service}-{operation}-{pattern}` — the `report_id` of a standing-performance report (`pattern` is `n-plus-one`, `query`, or `cache`).
- key `reviewer:apm:{service}` — a resolved owner (bare lowercase GitHub login) for a service, so reports route to a human faster.

### Decide

The generic report mechanics — search the inbox first (via the `report:apm:{service}:{operation}` pointer, else an `inbox-reports-list` search on the specific service / operation, not a broad word like `latency`), edit-vs-author, the status rules, reviewer routing, the retry and cross-run deduplication rules, and the `priority` / `repository` fields — live in the harness prompt and in `authoring-scouts` → `references/report-contract.md`. Do not re-derive them here. This section is only the APM judgment layered on top:

- **Edit** when a still-live report already tracks the operation — an error rate still stepped up, a p95 still elevated, a service still dark. A persistent regression is one report across runs: a new complete bucket confirming it is ongoing is a re-escalation (`append_evidence` with the fresh before/after numbers), not a fresh report per tick.
- **Author** when nothing live covers the regression. A report-worthy finding names the concrete `(service, operation)`, gives before/after numbers (rate or p95, with the steady denominator), dates the onset bucket, and explains the shape that rules out a volume explanation, with the query results in the `evidence`. Attach the regressing series via `charts` — the rate or p95 with the steady request volume alongside — so the rate-step-on-steady-denominator shape is visible. A cause you have not named yet is not human input: a regression that names the `(service, operation)` and the shape of its error or latency move gives an agent somewhere to start, so set `actionability=immediately_actionable` and name the `repository` that serves the operation (omit the field when you can't tell which repo that is, so selection can find it). Keep `actionability=requires_human_input` for a regression whose next step is a call only a person can make — a dependency you don't control, a capacity or cost trade-off — and say which call in the summary. Priority: an active error-rate regression hitting many requests is **P1**; a contained latency regression **P2**; a single-dependency or low-traffic operation **P3**.
- **Bundle correlated operations into one report.** When a cluster of operations in one service / subsystem regressed together, file one report for the shared cause, not one per operation — an inbox full of six reports for the same slow deploy is noise.
- **Standing performance issues are P3 by default, capped, and evergreen-deduped.** The rules are in `references/performance-issues.md`; at most 1–2 new reports from that lane per run, and never in place of a regression report.
- **Remember** if real but below the bar, or worth carrying forward (a fresh baseline, a blip to watch), or to record what you ruled out and why.
- **Skip** with a one-line note if a `noise:` / `addressed:` / `dedupe:` entry, or an existing inbox report, already covers it with no material change.

### Close out

One paragraph: which services/operations you scored, which reports you authored or edited, what you remembered (baselines, blips), what you ruled out (volume-tracking spikes, deploy blips, dev services), and whether the standing-performance audit ran and what it found. "Looked but found nothing meaningful" is a real outcome. Don't write a separate "run metadata" scratchpad entry — this summary already serves that role.

## Disqualifiers (skip these)

- **Raw count tracking traffic.** Error or span count up in lockstep with request `count` (rate ~flat) — volume, not a regression. This is the dominant false positive; check it first.
- **Deploy-window blips.** A one-bucket p95 or error spike that recovers on its own. Record a `noise:`/`pattern:` entry; report only when sustained across ≥2 complete buckets.
- **High-but-steady error baselines.** An operation erroring at the same elevated rate in both windows (e.g. ~98% now and ~98% a week ago) is a standing baseline, not a fresh regression — record it once in `pattern:`/`noise:` memory and don't re-report it each run. The signal is the rate _stepping up_, not its absolute level.
- **Dev / test services.** `service.name` or a resource attribute (`deployment.environment`, env) of `dev` / `local` / `test` / `staging`. Filter before weighing.
- **Health-check / readiness endpoints.** `/health`, `/healthz`, `/ready`, `/livez` and the like — high volume, low signal. Allowlist them in memory.
- **Cold-start / low-traffic noise.** A p95 jump on an operation with a tiny `count` (n too small for a stable percentile) is usually a cold start or a single slow trace, not a trend.
- **Transient client retries.** A `Client` span that errors but whose parent ultimately succeeds (retry succeeded) — don't report unless the failure rate itself is climbing.
- **Single-trace anomalies.** One slow or error trace with no recurrence across the window.
- **Known upstream provider / DB errors** already covered by memory — re-report only if the rate or shape changed meaningfully.

When in doubt, write memory instead of filing a report.

## MCP tools

Direct (read-only): `apm-service-health`, `apm-services-list`, `apm-spans-aggregate`, `apm-spans-sparkline`, `apm-spans-tree`, `apm-spans-duration-histogram`, `apm-attribute-breakdown`, `apm-attributes-list`, `apm-attribute-values-list`, `apm-spans-count`, `query-apm-spans`, `apm-trace-get`.

Inbox & reviewer routing (mechanics in `authoring-scouts` → `references/report-contract.md`):

- `inbox-reports-list` / `inbox-reports-retrieve` — the reports already in the inbox; check before authoring so you edit instead of duplicating.
- `inbox-report-artefacts-list` — a comparable report's artefact log; reviewer precedent.
- `scout-members-list` — the in-run roster for routing `suggested_reviewers` to a service owner.

Harness-level: `scout-project-profile-get`, `scout-scratchpad-search`, `scout-runs-list`, `scout-runs-retrieve`, `scout-emit-report` / `scout-edit-report` (author / edit a report — the report-channel contract is in the harness prompt), `scout-scratchpad-remember`, `scout-scratchpad-forget`. Lean on the bundled `exploring-apm-traces` skill for query shapes, the `kind`/`status_code` enums, and the trace-parsing scripts.

