Run Tests With Testomatio Reporter

Run Tests with Testomat.io Reporter

npx @testomatio/reporter creates test runs in Testomat.io and launches groups of tests — locally or remotely through a Testomat.io CI profile. Every command requires the TESTOMATIO env var (the project API key, tstmt_*) and exits 1 without it. Every value in angle brackets is a placeholder.

Pick the command by intent

Run kinds

--kind declares what the run contains:

A run created with start executes nothing: manual cases are pending immediately; an automated part stays scheduled until launched. --format id prints only the run id to stdout (banner and logs go to stderr), so capture is clean:

RUN_ID=$(npx @testomatio/reporter start --kind manual --format id)

What goes into the run

Without a filter, nothing is scoped:

• start creates a run with no predefined test list — results land in it later, when tests report with TESTOMATIO_RUN=<run-id>.
• run "<runner command>" executes the full suite and reports every result.
• run --remote <profile-name> dispatches the Testomat.io CI profile with its default scope.

--filter "<pipe>:<criteria>" narrows the run to the matching tests instead. Two filter pipes exist — testomatio: (match by test metadata stored in the project) and coverage: (match by changed source files); any other prefix is rejected. start and run accept the same filters, so a filtered run can be prepared first and launched later.

Filter by tag, plan, label, or Jira ticket (testomatio:)

• The value must match exactly — the tag name, plan id, label, or ticket as stored in the project.
• Works with every test framework whose runner accepts --grep (Playwright, CodeceptJS, Cypress, etc.) — the filter resolves to a grep pattern the runner consumes.

Filter by changed source files (coverage:)

--filter "coverage:file=<path-to-coverage-map>,diff=<git-ref>"

This is how "run only the tests affected by a code change" works — and it needs a coverage map: a YAML file mapping source files/globs to test IDs and tags. The reporter cannot know by itself which tests cover which code; the map provides that link, and the diff selects which of its entries are affected. Create the map with the qa-test-code-coverage skill (default coverage.tests.yml, one file serving both manual and automated tests).

• file= — path to the coverage map. May be absolute; it is read with fs, independent of the working directory.
• diff= — git ref to diff against; defaults to master. The reporter runs git diff <ref> --name-only in process.cwd() — launch it from inside the repo whose changes are being detected.
• Changed files are mapped through the YAML; the matching suite/test IDs and tags become the run's scope.
• Zero matching tests → no run is created (No tests found.). Exit codes: start and --filter-list exit 1, run exits 0; --warn makes all of them exit 0.

Picking the diff base

• Changes on a branch → diff against its target branch (e.g. origin/<default-branch>). Full git history must be available (fetch-depth: 0 or the CI's equivalent).
• After a merge the target branch equals HEAD, so diffing against it yields nothing — use the previous mainline tip: HEAD~1 for squash merges, HEAD^1 for merge commits.

List matching tests without running (--filter-list)

--filter-list resolves a filter and prints the matching test IDs — nothing executes, no run is created. --format picks the encoding: ids (comma-separated, default), grep (alternation pattern), json, newline. Exit code 0 when at least one test matched, 1 when nothing did — scripts can branch on it:

GREP=$(npx @testomatio/reporter run --filter-list "coverage:file=<coverage-map>" --format grep)
[ -n "$GREP" ] && npx playwright test --grep "$GREP"

Cannot be combined with --remote.

Name and group the run

• TESTOMATIO_TITLE — the run title (e.g. PR <number>: <title>).
• TESTOMATIO_RUNGROUP_TITLE — groups related runs (per week / release / milestone).
• TESTOMATIO_DESCRIPTION — free-form run description, shown on the run and in PR/MR comments; put the link to the change here (a generated coverage scope is appended after it).
• TESTOMATIO_ENV — optional environment labels.

Report into an existing run

• TESTOMATIO_RUN=<run-id> — the command reports or launches into that run instead of creating a new one. Works across pipelines and even across repos — pass it (with TESTOMATIO and the title env) into whatever process executes the tests.
• No id at hand → shared-run title matching. Set on both sides:
  • TESTOMATIO_SHARED_RUN=1 — match the run by TESTOMATIO_TITLE instead of creating a new one;
  • TESTOMATIO_TITLE — the match key, identical on both sides;
  • TESTOMATIO_SHARED_RUN_TIMEOUT — minutes the title stays matchable, default 20; size it to the expected gap between create and launch.

Remote launch (run --remote)

--remote <profile-name> asks Testomat.io to dispatch a Testomat.io CI profile — a CI workflow configuration saved on the project (Settings → CI) — instead of executing tests locally. Testomat.io triggers that workflow, and its results report back into the run:

TESTOMATIO_RUN=$RUN_ID npx @testomatio/reporter run --remote <profile-name> \
  --filter "coverage:file=<coverage-map>,diff=<git-ref>"

• With TESTOMATIO_RUN, the Testomat.io CI profile is triggered for that run; without it, a new run is created.
• Works with every filter form — the resolved test ids are forwarded to the CI workflow as a grep pattern. No filter → no grep; the workflow runs its default scope, or a prepared run's stored scope from creation time. A fresh --filter at launch replaces the stored scope.
• --remote-param <key>=<value> forwards a parameter to the Testomat.io CI profile config (repeat for several) — e.g. a preview URL or target branch.
• Guards: cannot combine with --filter-list; any positional command is ignored with a warning; a missing Testomat.io CI profile fails with CI launch failed: <message> and exit 1.

Choosing the Testomat.io CI profile

Profiles differ by workflow, job names, and parameters — never guess one. When talking to the user, always say "Testomat.io CI profile" in full and explain what it is; the bare word "profile" means nothing to them.

• Testomat.io MCP connected → fetch the Testomat.io CI profiles, present the list, and ask the user to choose (see testomatio-mcp to connect).
• No MCP → ask the user for the Testomat.io CI profile name.
• None exists yet → the user must create one in Testomat.io (Settings → CI) first.

Local execution (run "<runner command>")

Wrap the runner and results report into the run as they come. Without a filter the whole suite runs; with one, the filter generates the grep the runner consumes:

TESTOMATIO_RUN=$RUN_ID npx @testomatio/reporter run "<runner command>" \
  --filter "testomatio:tag-name=<tag>"

PR/MR comments

The reporter posts and updates the PR/MR summary comment itself — start posts a pending comment with the planned tests the moment the run is created, and results replace it as they report. Never script a comment API call. Enable the pipe by setting its token on every reporter command, start included:

No pipe for the platform (e.g. Azure DevOps) → no comment; results remain visible in Testomat.io.

Related skills

qa-test-code-coverage (creates the coverage map the coverage: filter needs), setup-change-aware-pr-testing (wires these commands into a CI pipeline), qa-e2e-tests-reporting (install and configure the reporter in an automation project).