Monitor Test Run

Overview

Given a testRunId, watch the run and narrate it. The skill:

1. Reads live_remediation_enabled once via getOrgSettings, so it can explain deterministically what happens when an execution is blocked.
2. Runs the bundled scripts/poll-test-run.js in the background — it watches the run and emits a line only when an execution's state changes (the model never hand-polls).
3. Reacts to those emitted lines: surfaces a blocker (with the live-remediation URL) when one appears, stays silent in between.
4. On the poller's DONE, does a post-mortem so a blocker is never mistaken for a pass.

The skill is conversational: its "output" is the messages it posts to the user (events + final summary), not a value returned to a caller. It changes nothing server-side except, optionally, terminateTestRun if the user asks to stop the underlying run.

Tool naming. This doc refers to Kobiton MCP tools by their bare names (getOrgSettings, terminateTestRun). The registered name depends on how the host loaded the MCP server (Claude Code as a plugin exposes mcp__plugin_automate_kobiton__getOrgSettings; a repo-local .mcp.json exposes mcp__kobiton__getOrgSettings; other hosts differ). Use the bare name and let the host resolve the prefix. (Run state is read by the bundled poller over REST, not via the MCP getTestRun tool — a background process can't call MCP tools.)

Prerequisites

• An authenticated Kobiton MCP connection. All three tools resolve the caller's org/user from the OAuth context.
• A testRunId — usually the one createTestRun just returned, or one the user names.
• getOrgSettings available. If the host can't find it, fall back to the flag-OFF branch (see Step 1) rather than failing the watch.

Inputs

Status vocabulary (read before Step 3)

getTestRun returns revisit_executions[], each with: { id, test_case_id, status, failure_type, execution_session_id, assigned_device_id, execution_error }. assigned_device_id is the device the execution runs on — use it directly to build the live-remediation URL (Step 4); no separate session lookup is needed.

Execution status values: NEW, SCHEDULED, RUNNING, UPLOADING_IMAGE, COMPLETED, TERMINATING, NETWORK_PAYLOAD_SCANNING. COMPLETED is the terminal status; what kind of terminal it is comes from failure_type.

failure_type values (the ones this skill keys on): NONE (clean), BLOCKER_ENCOUNTERED, TERMINATED_BY_USER, TERMINATED_BY_SYSTEM, and several genuine failures (NETWORK_ISSUE, INIT_SESSION_FAILED, APP_CRASHED, TARGET_ELEMENT_NOT_FOUND, OTP_CODE_ENCOUNTERED, …).

The blocked moment depends on the flag — this is the key behavior:

• Live remediation ON: when an execution hits a blocker it pauses in a blocked-waiting state (live status BLOCKED_WAITING, then BLOCKED_RESUMING once remediation is accepted) and waits for the user to resolve it interactively. It does not end yet.
• Live remediation OFF: there is no waiting state. The execution ends immediately — status = COMPLETED, failure_type = BLOCKER_ENCOUNTERED — and the user's later portal resolution applies on the next rerun.

So with the flag OFF you will typically never observe a blocked-waiting status: the first you see of the blocker is the terminal COMPLETED + BLOCKER_ENCOUNTERED. With the flag ON you see the pause first, then either a resume or (if the wait expires / is not resolved) a terminal COMPLETED + BLOCKER_ENCOUNTERED.

Steps

1. Set up once: flag, portal base, and (if flag ON) the open-on-blocker preference

Do all of this before the watch loop, so when a blocker hits later there is nothing left to decide.

1a. Read the live-remediation flag. Call getOrgSettings a single time and cache settings.live_remediation_enabled as flagOn for the rest of the run — do not re-read it on every poll. Fallback (safer default): if getOrgSettings errors or has no live_remediation_enabled, treat flagOn = false and tell the user once that you couldn't read the flag and are assuming OFF.

1b. Derive the portal base URL from the MCP server the run lives on — do not hardcode https://portal.kobiton.com. Read the MCP server URL from .mcp.json and map the api host to its portal equivalent (drop any trailing /mcp), same rule as run-automation-suite:

So a run watched through api-test.kobiton.com → https://portal-test.kobiton.com, api-test-green → portal-test-green, etc. Cache this as <portal>. If the mapping can't be resolved, fall back to https://portal.kobiton.com. Use the MCP server that serves getTestRun for this run — i.e. the env the run and its devices live on — not the env any cross-org flag read happened to use.

1c. State the flag state, and — if flagOn = true — settle the open-on-blocker preference up front.

• flagOn = false → "Live remediation is not enabled for your org — a blocked execution ends immediately with BLOCKER_ENCOUNTERED, and a resolution you submit in the portal applies on the next rerun." (Nothing to pre-arrange; there is no live window to open. autoOpen is irrelevant.)
• flagOn = true:
  • If autoOpen was already decided by the caller (e.g. the create-test-run skill delegated here after the user chose "monitor + auto-open" → autoOpen = yes, or "monitor only" → autoOpen = no), do not ask again — just state it: "Live remediation is enabled; I'll {auto-open the live-remediation window on a blocker / surface the URL on a blocker}." Skip straight to Step 2.
  • Otherwise (invoked directly, no pre-set preference) → "Live remediation is enabled — if an execution hits a blocker it pauses and waits for interactive remediation." Then ask now, once, before monitoring: "When a blocker hits, do you want me to automatically open the live-remediation window for the blocked device?" Cache the answer as autoOpen (yes/no). Asked a single time up front — not per blocker mid-run.

2. Stream the bundled poller — do NOT hand-poll

Do not write your own sleep/poll loop and do not narrate individual polls. That is exactly what made a prior run noisy — calling getTestRun over and over and posting "still NEW / still blocked, no change". The bundled poller emits a line only when an execution's state actually changes and exits when the run is done:

node $SKILL_DIR/scripts/poll-test-run.js --run-id <testRunId>

($SKILL_DIR is this skill's directory. Optional: --interval <sec> base cadence (default 3), --max-interval <sec> backoff cap (default 30), --max-errors <n> consecutive-error give-up (default 5), --waiting-heartbeat <sec> blocked-heartbeat cadence (default 60, 0 disables).) The script reads run state via REST using ~/.kobiton/.credentials (a background process can't call the MCP getTestRun tool — that's why this is a script). It diffs per execution, backs off during quiet stretches, and exits 0 on DONE.

Run it so each emitted line streams back to you as it happens — this is the whole point. A plain "background" launch is NOT enough: a backgrounded process's stdout does not re-engage you, so you'd sit idle and miss the blocker window (this was a real failure). Use the host's stream-a-command's-output mechanism so every line re-invokes you:

Whichever mechanism: you react only to the lines it prints. Stay silent between lines; there is no value in reporting "no change". The lines:

Short-circuit: if the run is already fully terminal, the script emits the terminal EVENTs then DONE on the first poll — same path, no special-casing needed.

The event kinds map to outcomes as follows (the script applies this; it's here so you narrate correctly): terminal_passed = COMPLETED + failure_type NONE; terminal_blocker_encountered = COMPLETED + BLOCKER_ENCOUNTERED (a blocker, never a pass); terminal_terminated = COMPLETED + TERMINATED_BY_USER/TERMINATED_BY_SYSTEM; terminal_failed = COMPLETED + any other failure_type.

4. Surface blockers (loud); stay quiet on passes

On an EVENT blocked line (flag ON, paused) or an EVENT terminal_blocker_encountered line (flag OFF, already ended) from the poller — post a message containing:

• Which execution (exec=) and, for a suite, which test case (tc=) is blocked.

• The device — the device= field on the event line (the poller reads it from assigned_device_id).

• The live-remediation URL — built from <portal> (the base from the poller's READY line, env-mapped, not hardcoded to production) and device=. Note: no &view=device-only — the full default device-launch view is wanted here so the Kobi AI chat panel and controls are visible alongside the device:

  <portal>/devices/launch?id=<device>
  

• A deterministic explainer keyed on the cached flagOn:

  • flagOn = true → this is an action request, not a status update — the execution is now waiting on the user, on a clock. Say so plainly and urgently, e.g.: "⏳ Action needed now — this execution is paused on a blocker and waiting for you. It will auto-fail with BLOCKER_ENCOUNTERED if the remediation window times out, so resolve it promptly. {If autoOpen = yes: "I've opened the live-remediation window —" else: "Open the URL to"} chat with Kobi AI / drive the device to clear the blocker; the execution resumes automatically once it's cleared." Do not frame this as "I'll just wait and ping you when something changes" — by the time the next change arrives it may already be the timeout.
  • flagOn = false → "Live remediation isn't enabled for your org, so this execution ended with BLOCKER_ENCOUNTERED. Submit your resolution in the portal — it applies on the next rerun."

If the event line shows device=- (the device id is unexpectedly absent — it is part of the getTestRun contract, so this points to a backend/state problem): surface the blocker text without a URL and note INVALID_EXECUTION_STATE (missing assigned_device_id). Don't fabricate a device id.

4a. Auto-open the live-remediation window (flag ON, per the up-front preference)

The decision was already made in Step 1c — do not ask again per blocker mid-run. On each EVENT blocked line:

• If autoOpen = no → just post the URL (the user opens it themselves). Done.
• If autoOpen = yes → open the live-remediation window for the blocked device now, automatically, then carry on monitoring. (Still print the URL too, as a fallback.)

To open it, reuse run-automation-suite's chromeless-launcher chain — the same chain drive-automation-session uses — do not re-implement it. Two differences from that skill's device-only live view:

1. Full view, not device-only. Use the URL without &view=device-only so the Kobi AI chat panel and the surrounding launch-view controls render, not just the phone screen.
2. Bigger-than-phone window. Size the window wide enough to fit the device plus the chat interface — default 1400×900 (vs the launcher's phone-shaped 540×920 default). The device sits on one side and the chat/controls on the other.

LIVE_REMEDIATION_URL="<portal>/devices/launch?id=<assigned_device_id>"

Invoke per host OS, in the background (Claude Code: Bash tool with run_in_background: true; other hosts: append & and disown) so the launcher's resize-polling doesn't block the watch loop. $SKILL_DIR is this skill's directory.

Launcher exit codes (surface in the background-task completion event): 0 = Chrome launched (resize may not have succeeded — informational); 2 = Chrome not found → fall back to telling the user to open the URL manually (or their default browser); 64 = usage error → surface it. The URL is always printed too, so a launcher failure never blocks the user.

This is only for the flag-ON branch — with the flag OFF the execution has already ended, so there's no live window to open; just give the explainer and the portal URL.

On a resumed event, post a concise "execution [id] resumed" line.

Stay quiet on in-flight passing progress — don't narrate every poll. But "quiet" applies to running executions, not to blocked-waiting ones: a flagOn = true blocker is the user's to clear against a timeout, so silence there is wrong (it reads as "nothing for me to do" while the clock runs out).

4b. While any execution is blocked-waiting, keep the pressure on (flag ON)

Track which executions are currently in the blocked state (entered on an EVENT blocked, cleared on that execution's resumed or terminal event). The poller drives the nudging for you: while that set is non-empty and nothing else changes, it emits a throttled WAITING blocked=<n> line (default every 60 s).

• On each WAITING line, post a short standing reminder naming what's still pending and that it will auto-fail on timeout — e.g.: "⏳ Still waiting on you: 2 executions paused on blockers (Pixel 8, Pixel 4). They auto-fail with BLOCKER_ENCOUNTERED if not resolved soon — resolve them in the open windows." Don't treat WAITING as a no-op.
• When a previously-blocked execution goes terminal as terminal_blocker_encountered, say it timed out / wasn't resolved in time (not merely "ended with a blocker") so the cause is unambiguous.

The point: a blocked-waiting run is an open ask of the user, not a background watch. Keep it visible until they act or it resolves.

5. Post-mortem + final summary

On the poller's DONE line, post a final summary from the terminal EVENT lines you collected. For each execution, report by its terminal kind:

• terminal_passed → "execution [exec] passed".
• terminal_blocker_encountered → Never report this as passed.
  • If flagOn = true and this execution had been blocked (paused) → it timed out: "execution [exec] hit a blocker, paused for live remediation, and timed out unresolved → BLOCKER_ENCOUNTERED. The resolution you submit in the portal applies on the next rerun." (Flag this if the user could still have acted — it's the avoidable case.)
  • If flagOn = false → "execution [exec] ended with BLOCKER_ENCOUNTERED (live remediation not enabled for your org); the resolution you submit in the portal applies on the next rerun."
• terminal_failed → "execution [exec] failed: [failure=…]".
• terminal_terminated → "execution [exec] terminated ([failure=…])".

Format: group by tc= (test case) when the run is a suite; flat list per execution otherwise.

6. Lifecycle

• Already terminal at invocation → the poller emits the terminal EVENTs then DONE on its first poll; go straight to the final summary.
• Natural exit → the skill ends when the poller prints DONE (every execution COMPLETED); the monitor stream ends on the poller's exit. There is no arbitrary wall-clock cap; the test run's own lifecycle is the bound. (On Claude Code you may still TaskStop the monitor task after DONE to free it, though it has already exited.)
• User cancels the watch → stop the poller stream (Claude Code: TaskStop the monitor task; other hosts: terminate the streamed process) and end cleanly. The underlying run keeps going. If the user wants to actually stop the run (not just the watch), call terminateTestRun(testRunId) and confirm.

Errors

Notes

• This skill is additive conversational glue. It reads the org flag (getOrgSettings), runs the bundled scripts/poll-test-run.js to watch run state (REST, emit-on-change — it does not mutate the run), optionally opens the live-remediation view in a browser (the shared chromeless-launcher, Step 4a) when the user opted in, and optionally stops the run (terminateTestRun). It does not itself drive the device or resolve the blocker — the user does that in the opened view — and it does not implement the live-remediation experience.
• The poller, not the model, owns the watch loop. Never hand-poll getTestRun in a sleep loop or narrate no-change polls — the poller exists precisely so the watch is quiet between real state changes.
• It composes with the test-run tools: createTestRun (start) → monitorTestRun (watch) → terminateTestRun (stop, if asked).
• The status values, failure_type values, and the blocker rule above are the Kobiton platform's contract for a revisit execution; if the platform changes them, update this skill rather than diverging.

Source: jeremylongshore/claude-code-plugins-plus-skills → plugins/testing/kobiton-automate/skills/monitor-test-run/SKILL.md

Also appears in: jeremylongshore/claude-code-plugins-plus-skills/plugins/testing/kobiton-automate/.codex/skills/monitor-test-run/SKILL.md