Vrt Regression Watch

vrt-regression-watch

The same vlmkit diff agent CLI exposes regression flags (--previous, --persist-summary, --no-history, --fail-on-regression). This skill explains how to wire them into a recurring loop, what state lives where, and what counts as a regression.

Invocation

The vlmkit CLI in this repo is invoked from source:

node --experimental-strip-types src/cli/vrt.ts <command...>
# e.g. node --experimental-strip-types src/cli/vrt.ts diff agent report.json --previous prior.json

The published binary (./dist/vrt.mjs or a globally installed vrt) may lag the source. If it rejects subcommands with Unknown command: diff, the dist is stale — run pnpm build or use the source form. All vrt ... invocations below assume one of these two forms.

When to use

• CI gate on every PR: "if this PR's diff is worse than main's reference, fail."
• Scheduled snapshot of a production URL: "alert me if the rendered output drifts."
• Long-running migration: "after each round, did we make forward progress or regress?"

When NOT to use

• One-shot diff with no history: vrt-visual-diff (use --no-history there if you want zero state).
• CSS auto-repair: vrt-css-fix-loop.
• First-time setup with no prior baseline: the first run can't detect regression — it just establishes the summary. Run twice.

How regression is detected

After each vlmkit diff agent run, the per-viewport diffRatio is written to .vlmkit/last-diff-for-agent.json. On the next run:

1. Load that file as the comparison summary.
2. Compare each viewport's current diffRatio against its prior value.
3. If the majority of viewports got worse (current > prior by a relative threshold), emit the ### ⚠ REGRESSION banner at the top of the Markdown.
4. If --fail-on-regression is set, exit 1.

The "majority of viewports" rule prevents a single jittery viewport from triggering false alarms.

Quickstart

--output <dir> is a directory path; vlmkit diff html writes <dir>/diff-report.json into it. Pass that JSON file path — not the dir — to vlmkit diff agent. (diff-report.json is also written as a legacy alias; both files have identical content.)

The filename is diff-report.json even when there's no migration involved because the writer is shared with vlmkit migration compare. Legacy name; tracked for rename in #50. Treat it as "the diff report" regardless of how you produced it.

A no-op re-run (same inputs both times) is guaranteed to produce no ⚠ REGRESSION banner — use this as a sanity check when wiring the workflow into CI.

REPORT=reports/diff-report.json

# First run: establishes baseline. No banner possible (no prior summary).
vlmkit diff html before.html after.html --output reports/
vlmkit diff agent "$REPORT" --persist-summary .vlmkit/baseline.json

# Subsequent runs: same baseline file for read AND write
# (local-rolling idiom — passing the same path to --previous and
# --persist-summary means "compare against last run, then overwrite").
vlmkit diff html before.html after.html --output reports/
vlmkit diff agent "$REPORT" \
  --previous .vlmkit/baseline.json \
  --persist-summary .vlmkit/baseline.json \
  --fail-on-regression

The default state path is .vlmkit/last-diff-for-agent.json; the example uses an explicit .vlmkit/baseline.json to show how to keep a stable reference (e.g. "diff against main's last good run, not against the PR's previous run").

State lifecycle

Two retention strategies:

• Local-rolling: let .vlmkit/last-diff-for-agent.json rewrite each run. Catches per-PR regressions ("did this commit make it worse than the previous commit").
• Branch-stable: commit a snapshot of the JSON (or store it in CI artifacts) for main, then have PR jobs compare against that fixed reference. Catches "did this PR drift main from a known good."

Per-PR CI gate pattern

# .github/workflows/visual-regression.yml (sketch)
- name: Restore main's baseline
  uses: actions/cache@v4
  with:
    path: .vlmkit/baseline.json
    key: vrt-baseline-${{ github.base_ref }}

- name: Render current PR + diff
  run: |
    vlmkit diff html main.html pr.html --output reports/
    vlmkit diff agent reports/diff-report.json \
      --previous .vlmkit/baseline.json \
      --persist-summary /tmp/pr-summary.json \
      --fail-on-regression \
      --out reports/diff.md

- name: Comment on PR
  if: always()
  run: gh pr comment ${{ github.event.number }} --body-file reports/diff.md

Why a throwaway --persist-summary path: in CI you don't want the PR-run's summary to clobber the cached main-summary. Pointing --persist-summary at a PR-specific tmp path (or anywhere outside the cache key) keeps the main reference clean. --no-history is NOT a substitute here — it skips load too, so --previous would be ignored and no regression detection would happen.

Flag reference

Default behaviour (none of the above): auto-load and auto-persist .vlmkit/last-diff-for-agent.json — i.e. local-rolling.

--no-history is the master switch — it skips BOTH load and write. Concretely: when --no-history is set, --previous and --persist-summary are ignored — no comparison happens, no summary is written. Pick one of these three modes per call:

Reading the banner

### ⚠ REGRESSION

3 / 4 viewports got worse vs. previous run:

| viewport | prior | current | Δ |
|---|---|---|---|
| mobile  | 0.012 | 0.084 | +0.072 |
| tablet  | 0.008 | 0.041 | +0.033 |
| desktop | 0.002 | 0.029 | +0.027 |
| wide    | 0.001 | 0.001 |  0     |

The Δ column is absolute. A viewport is "worse" if Δ > 0 and the relative increase exceeds the noise threshold (currently fixed in detectRegression). Fix order is usually mobile → up; mobile breakage is the most likely user-visible regression.

Combining with masks

The same --mask arg from vlmkit diff html applies. A regression banner appearing on a viewport you don't actually care about (e.g. wide) usually means you forgot to mask a flapping element on that viewport. Add the selector to --mask and re-run; if the banner disappears, the original signal was noise.

Failure modes

• First-ever run shows no banner → expected (no prior summary to compare). Run twice.
• Banner appears on every run, even with no changes → the page has unmaintained dynamic content; pass --mask for the flapping selectors.
• --fail-on-regression exits 1 but the banner says "0 viewports got worse" → there's a JSON-vs-Markdown drift bug; re-run with DEBUG_VLMKIT=1 and file an issue.
• Stale summary from months ago → manually remove .vlmkit/last-diff-for-agent.json (or your custom path) and re-run.

Environment

No additional env vars beyond what vlmkit diff agent needs (none for the pure pixel+CSD path).

Related skills

• vrt-visual-diff — same CLI, no state. Use first to confirm the diff looks sensible before wiring it into a watch loop.
• vrt-migration-eval — produces report.json files that vlmkit diff agent can consume just like the html-diff path.