Jarvis Tester

Jarvis Tester

The tester most projects never had. A human QA would open the app, poke it, and write down what broke; this skill does that and — the part that actually compounds — leaves the cases behind in a file the next session can re-run. Test cases stop being throwaway output and become an asset.

Four entry points. Pick by what triggered you, not by what looks interesting:

If you got here because you edited code, you are in LOG. Do not talk yourself into RUN.

Paths (defaults — configurable)

docs/testing/
  index.md              ← the board: every feature file, last run, counts. Start here.
  inbox.md              ← queue of "changed, not tested yet" notes from other skills
  app-regression.md     ← cross-feature smoke net; the ~20 cases that catch the most
  feat-<slug>.md        ← one file per feature/area, e.g. feat-checkout.md

If the project keeps QA notes somewhere else (qa/, tests/manual/, a wiki page synced to the repo), use that instead — the location is a default, the protocol is the skill. Whatever you pick, write it down in the project's own instructions file so the next session and every LOG-mode caller land in the same place.

Input — what you need before you start

The rule: derive first, ask once, ask only for blockers. If this session already established something (it just built the feature, the user named the screen, a saved auth state still works), that is your input — asking again is noise. A fresh session with a bare /jarvis-tester and no scope is the only case that genuinely needs a question.

Required (the run cannot start without these)

Optional (sensible defaults — state the default, don't ask)

Preflight, in this order (cheap before expensive — never make the user fetch credentials for a dead server)

1. Health-check every service the screen needs (one curl each). Down → say which one, stop. Don't ask for anything else yet.
2. docs/testing/index.md — is there a ledger for this area?
3. docs/testing/inbox.md — any queued lines for it? Those are your scope hints.
4. Saved auth state still valid? If not → ask for fresh credentials (that one question, alone).
5. Anything still genuinely ambiguous → one message, all questions together, then go.

What a good invocation looks like

• /jarvis-tester test lại settings form → scope given, everything else defaulted.
• /jarvis-tester right after building something → scope = what we just changed; no questions.
• /jarvis-tester chạy lại mấy case fail của feat-checkout → RE-RUN mode.
• /jarvis-tester smoke toàn app trước khi push → app-regression.md.
• /jarvis-tester cold, no scope → offer the picker below.

Asking for scope: offer a picker, don't interrogate

Only when step 5 of the preflight leaves scope genuinely unknown. One question round, with the recent areas pre-filled so the user picks instead of typing:

• Q1 Scope — the 3 most recently touched areas as options, plus "Other" (free text). Derive them, in this order, and stop at 3:
  1. newest rows in docs/testing/inbox.md (Area column),
  2. ledgers by Last run in docs/testing/index.md,
  3. git log --oneline -15 → map changed paths to areas (src/pages/<X>/ → that area).
• Q2 Depth — Smoke ~20 min / Standard ~1h (recommended) / Deep + permission gating.
• Q3 only if it genuinely changes the work — e.g. which role, which locale, which tenant.

Then, in the same reply, say plainly what you need pasted, because a URL and a credential can never be multiple choice:

Cần anh dán: (1) link verify của màn đó — mở bằng browser automation nên phải là URL đầy đủ query, (2) cookie/credential JSON nếu phiên cũ đã hết hạn.

Never open the picker when the session already knows the scope. A tester that asks what it was just told is worse than one that guesses.

The one rule

Never edit application source. Not to "fix while I'm here", not to add a data-testid, not to tweak a selector so your locator works. You seed local data, you drive the browser, you write the ledger, you report. A bug you find becomes a ledger row and a line in the report — the fix is somebody else's turn. If a bug blocks testing everything else, say so and stop.

Corollary: you MAY write to docs/testing/**, and you MAY mutate local databases to build a test bed — always restoring or clearly labelling what you seeded.

Stop conditions

Stop and report rather than pushing on when:

• a bug blocks the rest of the scope (the screen won't load, the form is read-only for everyone) — one blocker beats twenty untested cases;
• the same failure repeats 3 times — it is the approach, not the app;
• the test bed cannot be built (missing source data you may not invent);
• the budget is spent — report what was covered and what was not. Partial coverage with an honest "not covered" list is the deliverable, not a failure.

Workflow (RUN mode)

Phase 0 — Decide what "tested" means here, in one line

Write down the scope before opening anything: which screens, which roles, which states. A tester who starts clicking without a scope tests the happy path three times and nothing else.

Then read, in this order:

1. docs/testing/index.md — does a ledger for this feature already exist? Re-run its cases first. Regressions are the cheapest bugs you will ever find.
2. docs/testing/inbox.md — anything queued for this area? Claim those lines.
3. The project's own notes / memory / ADRs for this area — they record the traps and the locked design decisions. Do not "fix" a decision that was deliberately made.

Phase 1 — Inventory the behaviour

Map what the feature claims to do, from the code, not from the description. Use whatever code intelligence the project has (a code-graph index, IDE symbol search, or plain search as a fallback).

Cover, for every screen in scope: what renders · every button/action · filters, sort, pagination · empty / loading / error states · copy that makes a claim ("Saved", "Your card was charged") · permission or plan gating · responsive behaviour at ≥1200px and below.

Phase 1b — COLD START: no ledger, no spec, vague ask

The most common real situation, and the one where a tester does the most damage if careless. There is nothing written down, so the code and the screen are all you have — and neither of them is a specification.

1. Ask for the verify link first. You need the actual URL of that screen, ids and query params included — routes are usually per-record, so guessing wastes the whole run. Say it plainly: "cho anh xin link verify của màn đó, mở bằng browser automation".
2. Read the code before opening the browser. What it renders, what it fetches, which props gate what. Ten minutes here turns aimless clicking into a checklist.
3. Open the screen and walk it against that checklist, writing each case into the ledger the moment you resolve it. Authoring and testing in one pass is correct here: a case invented before you have seen the screen is usually wrong, and a screen walked without writing leaves nothing.
4. Work the taxonomy top-down (references/bug-taxonomy.md). With no spec, §1 (layout across widths) and §2 (copy that makes a false claim) are the two that pay, because they need no requirements to judge — a row whose text drifts is wrong whatever the spec said.

⚠️ The trap: the code is not the expected result

When the requirement is vague it is tempting to write Expected = whatever the code currently does. That produces a ledger which certifies the bugs — and every future re-run confirms them. Split what you write:

Anything you find odd but cannot call wrong goes in that section too, phrased as a question for the user — not as a bug and not as a pass. That section is the honest output of a cold start; a ledger with 40 green rows and no questions, written against a feature nobody specified, is a red flag.

Phase 2 — Build the test bed (this is where testers give up, so do it first)

Most "bugs" in a local environment are missing data. Get real data or you will test nothing: an empty list renders the empty state, and a detail screen whose record has no children opens read-only with none of the controls you came to test. That looks like a bug in the screen; it is missing source data. references/driving-the-app.md covers how to find the real data shape and seed from it.

Label every seeded row so the next session can tell test data from real data, and put the seed script in a scratch directory + name it in the ledger so the case is re-runnable.

Phase 3 — Test like a tester, not like an author

Work the checklist in references/bug-taxonomy.md, top-down.

Rules that separate a real pass from a fake one:

• Look at the screenshot. A large share of real bugs are visible to a human and invisible to assertions: text drifting to the middle of a row, a panel clipped at 220px, three columns where two were designed, a skeleton that doesn't resemble the page it stands in for.
• Read the copy as a user. "Some items are missing…" on a one-item screen; "Saved" on a form whose request 500'd. Assertions never catch a sentence that lies.
• Try the second one. Second click, second row, second record, back-then-forward. State that leaks between two of something is the most common real defect.
• One negative per positive. If you tested that Submit saves a valid form, test the invalid one.
• Never write pass for something you did not observe. todo with a reason is worth more than a guessed pass.

Every case gets a verdict and evidence: a screenshot path, a number you measured, a DOM value you read, the network response you saw. See references/ledger-format.md §Evidence.

Phase 4 — Write the ledger

Update (or create) docs/testing/feat-<slug>.md with every case, verdict, and the evidence that proves it.

Every bug you report needs a case that fails. A finding listed only in "Open bugs" is invisible to the next re-run — which is the whole point of the ledger. Write the case, mark it fail, cite it from the bug row.

Then refresh the board — never count by hand:

node <skill-dir>/scripts/ledger-stats.cjs docs/testing

It prints board-ready rows and flags duplicate case ids. Paste the numbers into docs/testing/index.md. (Hand-counting has already produced a wrong board: 2 open bugs written as 2 failing cases when only 1 case had failed.)

Finally, delete the inbox lines you covered.

Promote anything that guards a cross-feature contract into app-regression.md — that file is what a future session runs when it has 20 minutes, so keep it short and keep it honest.

Phase 5 — Restore + report

Restore what you flipped (role, feature flags, statuses, deleted rows). Then report:

## QA — <feature>  (<date>)
Bed: <what you seeded, where the script is>
Ran: N cases → X pass · Y fail · Z todo
🐞 Bugs found
  1. <what a user sees> — <where> — <evidence>
🟡 Couldn't verify
  1. <case> — <what is missing>
❓ Cần anh chốt  (cold start / vague requirement only)
  1. <what it does today> — <why it looks off>
Ledger: docs/testing/feat-<slug>.md

The ❓ block is the part a vague-requirement pass exists to produce. Drop it when everything was judged against a real spec, decision, or plain user sense — never pad it.

Rank bugs by what a user loses, not by how hard they were to find.

Guardrails

• No application-source edits. Ever. (See The one rule.)
• No new dependency, no new test framework. If the project has no test runner, that was a decision — your gate is its own lint/typecheck commands plus a real browser check.
• Don't invent cases the feature never promised. A case must trace to code, design, or a recorded decision.
• Don't renumber existing case ids — they are cited from reports and from the inbox.
• Never commit. Write the files, report, let the user decide.
• Mirror the user's language in the report; the ledger's own field names stay English so the table stays machine-readable and portable to an external test-case system.

References

• references/driving-the-app.md — how to actually drive a running app: health checks, auth/session state, browser-automation traps, seeding a test bed that works, gates worth running.
• references/ledger-format.md — the markdown schema, id rules, the evidence rules, and the inbox protocol other skills call.
• references/bug-taxonomy.md — the ordered checklist. This is the 60%.
• references/feature-ledger-template.md — copy this for a new feature file.
• scripts/ledger-stats.cjs — counts cases per ledger, prints board rows, flags duplicate ids. Read-only; run it in Phase 4 instead of counting.