Exploring Replay Vision observations
A scanner is a standing LLM probe over session recordings; each time it runs against a session it records
one observation. This skill is about the other half of the loop — reading what the scanners have found
and doing something useful with it. For creating or sizing scanners, use [[creating-replay-vision-scanners]].
Mental model
- Scanner → observations. One observation = one scan of one session. There is at most one observation
per
(scanner, session).
- The finding lives in
scanner_result. Its shape depends on the scanner's scanner_type, but it always
carries a confidence:
monitor → a verdict (yes / no / inconclusive) plus an open-ended observation.
classifier → one or more tags from the scanner's label set.
scorer → a numeric score on the scanner's scale.
summarizer → a free-text summary (optionally with facet embeddings).
- Only
succeeded observations carry a finding. Triage the rest by status/error_reason (see below).
- Observations are LLM judgments, not ground truth. One observation is one model's read of one session —
corroborate before you act on it.
If a scanner has emits_signals: true, its observations also feed the Signals pipeline and may surface as
Inbox signal reports (clusters of related findings). When the user's intent is "work the reports", that's
the inbox path — see Acting on findings below.
Step 1 — Anchor on the scanner
If the user gave a /project/<id>/replay-vision/<scanner-id> URL, that path segment is the scanner ID.
Otherwise list them with vision-scanners-list and pick the relevant one.
Then call vision-scanners-get to read its configuration before reading results — the scanner_type and
scanner_config.prompt tell you how to interpret scanner_result (a verdict field only makes sense once you
know it's a monitor; a score only means something against the scorer's scale).
Step 2 — Pull the observations
Pick the axis that matches the question:
- What has this scanner found, over time? →
vision-scanners-observations-list (the workhorse). Filter to
status=succeeded to get only sessions with a finding, then narrow by verdict (monitors) or tags
(classifiers). Scorers aren't filtered by score — rank them with order_by=-result_score instead. Use
order_by (e.g. -result_score, -completed_at) to surface the strongest hits first.
- What did every scanner find about one session? →
vision-observations-list (the session_id query
parameter is REQUIRED). Use this while investigating a single recording.
- The full detail of one finding →
vision-scanners-observations-get or vision-observations-retrieve —
returns the frozen scanner_snapshot (config at run time) and the complete scanner_result, including any
event citations that link the finding back to specific events in the recording.
Triage status so you don't mistake a non-result for "nothing wrong":
| status |
meaning |
typical error_reason |
succeeded |
has a scanner_result |
— |
ineligible |
session couldn't be analysed — a normal outcome, not an error |
too_short, no_recording, too_inactive, too_long, no_events |
failed |
the scan errored |
provider_rejected, validation_failed, rasterization_failed, provider_transient, internal_error |
pending / running |
still in flight |
— |
A scanner that looks like it "found nothing" is often producing mostly ineligible observations — check the
mix before concluding.
Step 3 — Read the findings
- Monitors: focus on
verdict: yes; treat inconclusive as a weak signal. The observation text is the
substance.
- Classifiers: group by
tags to see the distribution of what's happening across sessions.
- Scorers: look at the tails (highest/lowest scores), not just the average.
- Summarizers: read for recurring themes across summaries.
Weight by confidence, and don't over-index on a single observation. To understand a specific hit, take its
session_id and either cross-reference other scanners (vision-observations-list) or drill into the actual
recording with the [[investigating-replay]] skill and the session-recording MCP tools.
To test a scanner's lens against a specific session that doesn't have an observation yet, trigger one on demand
with vision-scanners-scan-session — it's async (minutes; rasterising the recording + the LLM call are slow)
and, like all observations, runs at most once per (scanner, session).
Step 4 — Act on the findings
Match the action to the user's intent, and corroborate before you create work:
- Summarize a pattern. Report the finding back with the numbers and a few representative
session_ids
(e.g. "12 of 40 succeeded observations flagged checkout confusion; sessions A, B, C"). Cite, don't assert.
- Make it trackable. When a finding is corroborated across several sessions (not one low-confidence
hit), capture it durably with the tools that exist: create an
insight or notebook to track its
frequency, bundle the supporting recordings into a session-recording playlist so a human can watch the
evidence, and add an annotation if it marks a regression. There is no MCP tool to open a PostHog
task directly — to route a finding into tracked work, use the Inbox path below (for signal-emitting
scanners) or hand the summary to a human or coding agent to act on. Group by distinct issue, not per
observation.
- Work the Inbox. If the scanner emits signals, its findings may already be clustered into signal reports —
read and act on those with
inbox-reports-list + inbox-report-artefacts-list (the report's work log is the
evidence). See the [[inbox-exploration]] skill; that path also records your work against the report.
The discipline that matters: a single observation is one model's judgment on one recording. Confirm a finding
reproduces across observations (or against the raw recording) before turning it into a task, an alert, or a
claim — the same rigor the signals pipeline applies before it promotes observations to a report.
Gotchas
- Only
succeeded observations have a scanner_result — everything else is triage metadata.
ineligible ≠ failed. Ineligible is a normal terminal outcome (e.g. the recording was too short), not
a bug to chase.
- One observation per
(scanner, session) — re-scanning a session that already has any observation
(even ineligible/failed) is a no-op.
- Findings are snapshotted. Each observation keeps the
scanner_snapshot it ran under, so older
observations may reflect a previous prompt/config (scanner_version).
- Quota is shared. On-demand scans count against the org's monthly budget — check
vision-quota-retrieve
before triggering a batch of them.
1---2name: exploring-replay-vision-observations3description: "Guides agents through pulling a Replay Vision scanner's observations, reading the findings, and acting on them — summarizing patterns across sessions, drilling into individual recordings, and turning real, corroborated issues into PostHog tasks, insights, or an investigating-replay hand-off.\nTRIGGER when: user wants to pull/read/triage Replay Vision observations, asks \"what has my scanner found\", wants to act on or summarize scanner findings, turn observations into tasks/work, or points at a /replay-vision/<scanner-id> URL.\nDO NOT TRIGGER when: creating or sizing a scanner (use creating-replay-vision-scanners), running a one-off scan you don't then analyse, or authoring a signals scout."4metadata:5 source: https://skillmd.com/skills/agentcorp-inc/exploring-replay-vision-observations6---78# Exploring Replay Vision observations910A scanner is a standing LLM probe over session recordings; each time it runs against a session it records11one **observation**. This skill is about the other half of the loop — reading what the scanners have found12and doing something useful with it. For creating or sizing scanners, use [[creating-replay-vision-scanners]].1314## Mental model1516- **Scanner → observations.** One observation = one scan of one session. There is at most one observation17 per `(scanner, session)`.18- **The finding lives in `scanner_result`.** Its shape depends on the scanner's `scanner_type`, but it always19 carries a `confidence`:20 - `monitor` → a `verdict` (`yes` / `no` / `inconclusive`) plus an open-ended observation.21 - `classifier` → one or more `tags` from the scanner's label set.22 - `scorer` → a numeric score on the scanner's `scale`.23 - `summarizer` → a free-text summary (optionally with facet embeddings).24- **Only `succeeded` observations carry a finding.** Triage the rest by `status`/`error_reason` (see below).25- **Observations are LLM judgments, not ground truth.** One observation is one model's read of one session —26 corroborate before you act on it.2728If a scanner has `emits_signals: true`, its observations also feed the Signals pipeline and may surface as29Inbox **signal reports** (clusters of related findings). When the user's intent is "work the reports", that's30the inbox path — see _Acting on findings_ below.3132## Step 1 — Anchor on the scanner3334If the user gave a `/project/<id>/replay-vision/<scanner-id>` URL, that path segment is the scanner ID.35Otherwise list them with `vision-scanners-list` and pick the relevant one.3637Then call `vision-scanners-get` to read its configuration **before** reading results — the `scanner_type` and38`scanner_config.prompt` tell you how to interpret `scanner_result` (a `verdict` field only makes sense once you39know it's a monitor; a score only means something against the scorer's `scale`).4041## Step 2 — Pull the observations4243Pick the axis that matches the question:4445- **What has this scanner found, over time?** → `vision-scanners-observations-list` (the workhorse). Filter to46 `status=succeeded` to get only sessions with a finding, then narrow by `verdict` (monitors) or `tags`47 (classifiers). Scorers aren't filtered by score — rank them with `order_by=-result_score` instead. Use48 `order_by` (e.g. `-result_score`, `-completed_at`) to surface the strongest hits first.49- **What did every scanner find about one session?** → `vision-observations-list` (the `session_id` query50 parameter is REQUIRED). Use this while investigating a single recording.51- **The full detail of one finding** → `vision-scanners-observations-get` or `vision-observations-retrieve` —52 returns the frozen `scanner_snapshot` (config at run time) and the complete `scanner_result`, including any53 event citations that link the finding back to specific events in the recording.5455Triage `status` so you don't mistake a non-result for "nothing wrong":5657| status | meaning | typical `error_reason` |58| --------------------- | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |59| `succeeded` | has a `scanner_result` | — |60| `ineligible` | session couldn't be analysed — a normal outcome, not an error | `too_short`, `no_recording`, `too_inactive`, `too_long`, `no_events` |61| `failed` | the scan errored | `provider_rejected`, `validation_failed`, `rasterization_failed`, `provider_transient`, `internal_error` |62| `pending` / `running` | still in flight | — |6364A scanner that looks like it "found nothing" is often producing mostly `ineligible` observations — check the65mix before concluding.6667## Step 3 — Read the findings6869- **Monitors:** focus on `verdict: yes`; treat `inconclusive` as a weak signal. The observation text is the70 substance.71- **Classifiers:** group by `tags` to see the distribution of what's happening across sessions.72- **Scorers:** look at the tails (highest/lowest scores), not just the average.73- **Summarizers:** read for recurring themes across summaries.7475Weight by `confidence`, and don't over-index on a single observation. To understand a specific hit, take its76`session_id` and either cross-reference other scanners (`vision-observations-list`) or drill into the actual77recording with the [[investigating-replay]] skill and the session-recording MCP tools.7879To test a scanner's lens against a specific session that doesn't have an observation yet, trigger one on demand80with `vision-scanners-scan-session` — it's async (minutes; rasterising the recording + the LLM call are slow)81and, like all observations, runs at most once per `(scanner, session)`.8283## Step 4 — Act on the findings8485Match the action to the user's intent, and **corroborate before you create work**:8687- **Summarize a pattern.** Report the finding back with the numbers and a few representative `session_id`s88 (e.g. "12 of 40 succeeded observations flagged checkout confusion; sessions A, B, C"). Cite, don't assert.89- **Make it trackable.** When a finding is corroborated across several sessions (not one low-confidence90 hit), capture it durably with the tools that exist: create an `insight` or `notebook` to track its91 frequency, bundle the supporting recordings into a session-recording playlist so a human can watch the92 evidence, and add an `annotation` if it marks a regression. There is **no MCP tool to open a PostHog93 task directly** — to route a finding into tracked work, use the Inbox path below (for signal-emitting94 scanners) or hand the summary to a human or coding agent to act on. Group by distinct issue, not per95 observation.96- **Work the Inbox.** If the scanner emits signals, its findings may already be clustered into signal reports —97 read and act on those with `inbox-reports-list` + `inbox-report-artefacts-list` (the report's work log is the98 evidence). See the [[inbox-exploration]] skill; that path also records your work against the report.99100The discipline that matters: a single observation is one model's judgment on one recording. Confirm a finding101reproduces across observations (or against the raw recording) before turning it into a task, an alert, or a102claim — the same rigor the signals pipeline applies before it promotes observations to a report.103104## Gotchas105106- **Only `succeeded` observations have a `scanner_result`** — everything else is triage metadata.107- **`ineligible` ≠ `failed`.** Ineligible is a normal terminal outcome (e.g. the recording was too short), not108 a bug to chase.109- **One observation per `(scanner, session)`** — re-scanning a session that already has any observation110 (even ineligible/failed) is a no-op.111- **Findings are snapshotted.** Each observation keeps the `scanner_snapshot` it ran under, so older112 observations may reflect a previous prompt/config (`scanner_version`).113- **Quota is shared.** On-demand scans count against the org's monthly budget — check `vision-quota-retrieve`114 before triggering a batch of them.