Test Features

Test Features

A feature that "should work" and a feature that has actually been driven through its real behavior, its edge cases, its data, and its neighbors are two different claims. This skill is how the second claim gets made — deliberately, in the same order every time, so nothing gets skipped because it seemed obvious.

Scope: decide what's actually being tested

• Explicit argument given (/test-features <description>, or the user names a feature in chat): that description is the scope. Still translate it into the real files/behaviors it corresponds to before testing — don't test your guess at what they meant, test the actual code.
• No argument: infer the scope from the just-finished coding session. Look at what actually changed — a diff against a sensible base, recently modified files, whatever the repo's version control shows — and cross-check that against the conversation's own record of what was built. Don't rely on memory of the conversation alone; the diff is ground truth for what changed, the conversation is ground truth for why and what it's supposed to do.
• Ambiguous scope — several genuinely distinct changes landed, or it's unclear which one the user means, or "the feature" spans more than can reasonably be one testing pass — do not guess. Ask via AskUserQuestion, listing the distinct candidates you actually found so the user is picking from real options, not naming one from scratch.
• Once decided, state the scope back in one line before starting, so it's on record for the report at the end.

Orient before testing anything

• Read what actually changed, and anything in the project that records intent or invariants for it (a CLAUDE.md, a progress/decision log, a README, comments at the change site). "Works correctly" is measured against what was actually intended, not against a guess.
• Identify the feature's real inputs, outputs, and side effects: what does it read, what does it write, what does it call. Identify what else touches that same data, function, or code path — this is the regression surface for the later step that checks nothing else broke.
• Identify what's actually available to test with, rather than assuming: an automated test/build/ lint suite (check package.json/Makefile/CLAUDE.md for the real commands, don't guess a generic one), a way to run the app locally, a deployed/live version, browser automation, database access. What's missing here shapes what step 6 below can actually do.

Whenever a step below finds a bug

• Small, obviously safe to fix (a clear, local, one-shot fix; no design ambiguity; doesn't touch data safety or need a judgment call) — fix it inline, right away, as part of this session. Re-run whatever check found it (and the full baseline suite if the fix touched shared/core code) to confirm the fix actually holds before moving on. Note it in the final report as found and fixed, with what the bug was.
• Anything bigger — the right fix isn't obvious, it touches architecture or data safety, or it needs a decision only the user can make — do not fix it. Report it clearly enough to act on (what's wrong, how to reproduce it, where it lives) and move on with the rest of the pass. Testing and fixing stay separate for anything past the "small and obvious" bar.

1. Run the project's own automated suite first — the baseline gate

Before any manual testing, run whatever the project's real test/typecheck/build/lint commands are (from Orient, above). This has to be clean — or its pre-existing failures explicitly called out as pre-existing, not new — before anything found manually means anything. A red baseline makes every later finding ambiguous. Report the exact commands run and their real output/exit codes, never "tests pass" from memory or from having run them earlier in the session.

2. Functional correctness — does it do what it's supposed to?

Walk every distinct behavior the feature has, one at a time, confirming each actually works — starting with the straightforward case, then everything the feature is explicitly supposed to prevent (a validation rule, an error path, an access restriction). Confirm the prevention actually happens; don't stop at the happy path.

3. Data correctness — verify the state itself, not just the response

If the feature reads or writes data (a database row, a file, a cache, another service's state), check the data itself after the action — not only whatever success message or response came back. A "200 OK" or a "confirmed" is not proof anything was written correctly. Check this holds under the conditions a real user's data would actually create: pre-existing records, several related rows, an empty/first-time state — not only a freshly-seeded, artificially clean one.

4. Edge cases and stress-testing like a real user would

Think concretely about what a real person would actually try — not an abstract checklist executed mechanically for its own sake. Pick the cases that are real for this feature:

• Unusual, extreme, or malformed input — empty, very long, wrong type, special characters, boundary numbers/dates.
• Doing things out of the expected order, twice, very fast, or starting then abandoning it mid-way.
• Doing it when something related is already in an unusual state — already deleted, already at a limit, being changed by something else at the same time.
• Doing it as more than one user or session at once, if that's possible for this feature.
• Deliberately bad or hostile input, if the feature accepts anything a user — or an attacker — controls.

Say which cases were actually tried and what happened for each; a list of ideas that weren't run isn't testing.

5. Regression — did this break anything else?

Beyond the automated suite in step 1, specifically re-check the other features or paths identified in Orient that share code, data, or a surface with the change — this is exactly what the automated suite misses if there's a coverage gap there. If the change touched something widely shared (a core loop, a common function, a schema), say explicitly what was checked and why those were the right things to check, not just that "nothing else seemed affected."

6. Live / real-environment test — as an actual user would experience it

If there's a running dev server, a deployed environment, or a browser-usable UI, exercise the feature there — not only at the code or API level — since this is the only way to catch what only shows up in real usage (rendering, timing, a real multi-step flow). Use browser automation if it's available and fits the feature; if the feature is backend/API-only, exercise it the way its real caller actually would, not by calling the function directly.

If exercising it this way needs an actual write or state change against a live environment holding real user data, stop and ask the user how to proceed before doing it. Don't decide unilaterally between a dev copy and the real thing, and don't skip it silently — describe exactly what write is needed and why, and let the user decide (a dev/throwaway copy if one exists, explicit go-ahead for the real write, or skipping that specific check). This is a live decision every time it comes up, not a fixed policy — it's the same standing rule that a real write always needs explicit permission, applied here.

If no live or deployed environment exists or is reachable at all, say so plainly rather than letting this layer silently disappear from the report.

7. AI/agent-specific checks — run whenever the feature touches model or agent behavior

Whenever what's being tested involves an LLM call, a tool/function call a model makes, more than one agent communicating, or content pulled in from somewhere (a document, a database, another agent's output) that then influences what happens next — add these on top of everything above, not instead of it:

• No unverified claims. Did the model state a fact, number, or claim as true that wasn't actually produced by a tool or a real data source — including a plausible-sounding invented one?
• Correct tool use. Did it call the tool(s) it was actually supposed to, with sane, correctly typed arguments — and does a bad or missing argument fail loudly rather than silently doing the wrong thing?
• Content it reads is not content it obeys. If the model/agent reads something a user or outside source authored (a saved note, a fetched page, another agent's report, a retrieved document), confirm instructions embedded in that content are NOT followed. Plant an obvious one (a line telling it to do something unrelated) and confirm it's treated as data, not a command.
• Multi-agent correctness, when more than one agent/subsystem is involved: does each stay in its own lane (never act on data or requests outside its own actual domain), and does the result stay correct when they run concurrently, or when one of them is slow or fails outright?
• Non-determinism awareness. If the same input can reasonably produce a differently-worded but equally correct response, don't treat wording differences as a failure — check the underlying claim or behavior, not the exact phrasing.

8. Coverage gaps

While testing — especially during steps 4 through 7 — if you find real behavior with no automated test protecting it at all, note it specifically (not a vague "could use more tests here"). At the end, list every such gap found and ask the user whether to write the missing test(s) now. Don't write them unprompted, and don't quietly drop them from the report either.

Reporting

Close with one structured report:

• Scope — what was actually tested, stated plainly.
• Baseline — the exact commands run in step 1 and their real result.
• Functional + data correctness — what was walked through, what held up.
• Edge cases tried — each one, and what happened.
• Regression check — what else was verified and why those were the right things.
• Live test — what ran where, or why it didn't happen (no environment, or a live write that needed permission and wasn't given).
• AI-specific checks — run or not, and why.
• Bugs found — fixed-inline ones named as fixed; bigger ones reported with enough detail to act on.
• Coverage gaps — listed, with the question of whether to write the missing tests.

State coverage honestly. If something couldn't be tested — no live environment, an ambiguous case, a write that needed permission that wasn't granted — say so explicitly rather than letting it read as "all clear." A pass that finds nothing wrong is a genuinely useful, real outcome; a step skipped because it seemed unnecessary is not the same thing as a step run that found nothing.

What this skill is not

• Not a substitute for building out the automated test suite itself — it finds real gaps and asks before filling them, never assumes that's wanted.
• Not a rubber stamp, and not a fix-everything pass — past the "small and obvious" bar in the bug- handling rule above, every real finding goes back to the user as a decision, not a unilateral fix.