E2e Pr Stabilizer

E2E PR Stabilizer

Stabilize the Playwright E2E suite for a single pull request using evidence, not assumptions. Spans, traces, and the live app are the source of truth — not the CI dashboard. This skill never proposes a fix without a measurement to point at, and never commits a fix until three consecutive local runs prove it works.

This SKILL.md is a thin index. Detailed procedures live in rules/*.md and templates/*.md. Each phase loads only what it needs.

What this skill combines

This skill is the orchestrator over those.

External dependency — the Playwright Healer agent. The healer methodology this skill drives fixes through is not an agent in this repo; it is Playwright's own healer (Playwright Test Agents: planner / generator / healer) running on the Playwright MCP server. Set it up with npx playwright init-agents --loop=claude (Playwright ≥ 1.56). At runtime, Phase 5 uses the healer when the Playwright MCP is connected (mcp__playwright__* tools present) and falls back to the inline root-cause methodology when it isn't — so the skill works with or without it, but is strongest with it. It does not duplicate their content — each phase delegates.

Modes

stabilize is the default because optimization edits (tightening timeouts, removing waits) carry flake risk that warrants human judgment. optimize runs Phases 1–4 only and emits a ranked recommendations report.

Input

$ARGUMENTS is parsed as [mode] [pr-ref] in any order:

• optimize (literal token) selects optimize mode; anything else is treated as pr-ref.
• pr-ref is a PR URL (https://github.com/<org>/<repo>/pull/13319) or PR number (13319).
• If pr-ref is missing, auto-detect the open PR for the current branch (same path as /ci-auto-fix Step 0).
• If mode is missing, default to stabilize.

Resolve mode + PR before doing anything else. See rules/input-resolution.md.

Workflow

Eight phases. Do not skip a gate. Phases 5, 6, and 7 are skipped in optimize mode (the Modes column says so explicitly).

Inner iteration in stabilize mode is local and bounded — see rules/local-iteration.md. The CI step in Phase 7 runs once. If CI disagrees with the local result, that is a signal to escalate, not to re-enter the loop blindly.

Required reading by phase

Load on demand. Do not preload.

For trace mechanics (zip → JSONL → action timeline), defer to /playwright-trace-analyzer. Do not re-implement.

Core principles

1. Iterate locally, ratify on CI. The inner loop is local because it is seconds-per-run, has the same trace artifacts, and emits the same OTel spans. CI runs once at the end as ground truth.
2. Data first, hypothesis second. Every fix is anchored to (a) a span with a measured failure rate, or (b) a trace action with a measured dur. "I think this is flaky" is not a finding.
3. Two evidence layers, not one. Spans tell you which tests fail and how often across runs (historical and local). Traces tell you why one specific run failed. A fix is only credible when both layers agree.
4. Validate empirically, not predictively. Confidence in a fix comes from running it, not from scoring it on paper. The two gates the skill enforces are both deterministic: selectors must exist (verifiable against source or live app), and the fixed test must pass three consecutive local runs (verifiable from exit codes).
5. Selectors must exist before they are used. Every new locator in a draft fix is verified against the component source (static) and, when ambiguous, the running app (live). The skill never commits a fix that names an element that does not exist.
6. Three consecutive local passes or no commit. A single passing run can be a coin flip on a flake. Three in a row — hardened with --repeat-each=3 when the measured flake rate is low (see rules/local-iteration.md) — filters out most unfixed flakes before we spend a CI cycle. The streak is necessary but not sufficient evidence; Phase 7's CI ratification plus the telemetry comparison is the real confirmation.
7. Never weaken the suite. No .skip, .fixme, waitForTimeout, continue-on-error, --no-verify, or removed assertions. The full list lives in rules/guard-rails.md.
8. Test-side fix unless the trace proves otherwise. Most flakes are selector, timing, or state-management bugs in tests. If the trace evidence points to product code, surface it to the user as a separate recommendation — do not silently mutate app code.
9. One PR at a time. Cross-PR refactors belong in a different skill.

Self-Improvement

/e2e-pr-stabilizer gets better across runs through a two-tier lessons loop (fast episodic tier + gated promotion), like autonomous-workflow and fix-bug. In stabilize mode it reads e2e-pr-stabilizer-lessons at Phase 4 (biasing the P1–P6 pattern classification and the Phase 5 locator strategy) and writes at Phase 7 — gated on the telemetry ratification signal, not the local 3-pass streak, so only fixes that CI actually confirmed fixed accrue a working lesson. The two tiers split naturally: the global scope holds universal race-shape → fix-shape mappings, the repo::{owner}/{repo} scope holds app-specific locator robustness. Lessons are advisory — they never relax a guard-rail, an empirical gate, or the 3-consecutive-pass requirement. A recurring lesson (seen_count >= 3) is promotion-eligible via /create-skill diagnose e2e-pr-stabilizer.

On the Phase 7 ratified verdict it also contributes a flaky hotspot to the shared codebase-knowledge bucket (hotspot::<test.file> counter) — a fact no other host produces, so every later code-changer plans around a known-unstable file. Write only, merge-never-clobber, on the ratified SHA (contract: ../../../agents/shared/rules/codebase-knowledge.md; mechanics in the loop file's cross-bucket write section).

optimize mode skips the loop (no fix, no ratification signal). Lessons run through LoreKit's memory.* tools (the lorekit-memory skill); if LoreKit is not connected the loop is a silent no-op. Full contract: rules/self-improvement-loop.md.

Anti-patterns

One-liners; the full list lives in rules/guard-rails.md.

• Pushing a fix because "the diff looks right" without three consecutive local passes.
• Trusting a single local pass as proof — flakes pass once routinely.
• Drafting a fix that uses getByTestId('foo') when nothing in the component source emits data-testid="foo".
• Patching waitForTimeout(1500) to mask a race instead of fixing the wait condition.
• Marking a test .fixme() because it is "flaky" without a measured cause.
• Treating a single failed CI run as evidence — flakes are statistical, so fetch the span history.
• Re-running CI hoping for a green without applying a code change.
• Editing product code based on speculation when the trace points at a selector or test-state issue.

Quickstart

/e2e-pr-stabilizer                                                       # stabilize, auto-detect PR
/e2e-pr-stabilizer 13319                                                 # stabilize PR 13319
/e2e-pr-stabilizer https://github.com/<org>/<repo>/pull/13319            # stabilize via URL
/e2e-pr-stabilizer optimize                                              # optimize, auto-detect PR
/e2e-pr-stabilizer optimize 13319                                        # optimize PR 13319

Once invoked, the skill drives end-to-end:

1. Resolves the mode and the PR.
2. Queries the Dash0 MCP for E2E spans filtered to this PR (git.pull_request_link) — historical baseline.
3. Reproduces locally with --trace=on, capturing trace.zip and (where the local OTel reporter is wired) fresh spans.
4. Correlates and produces an evidence-anchored finding set.
5. stabilize: drafts each fix, verifies every new locator resolves against source or the live app, then commits locally.
6. stabilize: runs the fixed test locally until it passes 3 times in a row (per fix).
7. stabilize: pushes once; watches the CI run; compares fresh telemetry to baseline.
8. Emits the report — stabilization (before / after + local-pass log + CI verdict) or optimization (recommendations).

Definition of Done

Both modes

• Mode (stabilize | optimize) and PR target resolved and printed.
• Historical telemetry pulled from the Dash0 MCP using the documented filter set, grouped by test name.
• Each queued test reproduced locally with --trace=on; trace artifacts captured per run.
• Each candidate has a span-side signature and a trace-side hotspot.
• Report written using the template, with the mode stated and findings ranked by measured impact.

stabilize only

• Every new locator in every applied fix was verified against the component source or via a live locator.count() ≥ 1 probe.
• Every applied fix passed 3 consecutive local runs with --trace=on and no failures or flakes within the streak.
• Fixes committed locally, pushed in one push, CI run watched to conclusion.
• Fresh telemetry pulled and compared to baseline — failures eliminated, retry counts reduced.
• No .skip, .fixme, waitForTimeout, or continue-on-error introduced (guard-rails check passed).

optimize only

• No commits, no pushes, no edits to test files.
• Each recommendation cites an estimated wall-clock saving (ms) based on the trace evidence.