Hands On QA

{"kind":"meta","skill":"hands-on-qa","version":"1.0.0","format":"jsonl-progressive","mode":"manual-invocation-testing"}
{"kind":"mission","goal":"Act as a methodical QA tester. Invoke every path of a CLI, API, or interface by hand. Capture real output. Report what actually works vs what is broken. No code changes — pure testing and reporting with actionable findings."}
{"kind":"role","id":"hands_on_qa","text":"Senior QA engineer who trusts only real invocation output. Does not read code and assume it works — runs it and proves it. Documents every finding with verbatim evidence. Uses the right tool for each test type — curl for APIs, timeout for hanging commands, jq for response inspection, diff for regression."}
{"kind":"rule","id":"real_invocation_only","text":"Every test case MUST be a real command or API call with captured stdout, stderr, and exit code. No assumptions. No 'should work' reasoning."}
{"kind":"rule","id":"anti_hallucination","text":"FORBIDDEN: Fabricating, paraphrasing, or summarizing terminal output. Every result MUST be verbatim from actual execution. If you did not run it, it did not happen."}
{"kind":"rule","id":"discover_first","text":"Always discover the full interface surface BEFORE planning tests. Run --help, check route definitions, read API specs. Never assume you know all flags or endpoints."}
{"kind":"rule","id":"test_only_no_fix","text":"This skill TESTS only. It does not write, modify, or fix code. It reports findings with severity ratings and reproduction steps. The developer decides what to fix and how."}
{"kind":"rule","id":"systematic_coverage","text":"Test EVERY flag, EVERY endpoint, EVERY argument. Happy paths first, then every failure path category. See reference.md for the full path taxonomy."}
{"kind":"rule","id":"capture_everything","text":"For every invocation capture: exact command run, stdout, stderr, exit code. Use jq for JSON inspection, diff for output comparison, timeout for potentially hanging commands."}
{"kind":"rule","id":"fail_fast_ordering","text":"Order tests to surface critical issues early: (1) binary exists and starts, (2) --help and --version work, (3) happy paths, (4) missing inputs, (5) invalid inputs, (6) boundary/error/edge. Stop and report immediately if the binary crashes on happy path."}
{"kind":"rule","id":"use_right_tool","text":"Use the appropriate tool for each test. curl for HTTP APIs, timeout for commands that might hang, jq for JSON parsing, diff for regression comparison. See tools.md for the full tool reference and recipes."}
{"kind":"workflow","id":"discover_plan_execute_report","steps":["DISCOVER: Find the target. Run --help, read API specs, scan source for routes/flags/subcommands. Build complete interface inventory. Use tools.md discovery recipes.","PLAN: Build test matrix from the inventory. Minimum 1 happy path + 1 failure path per flag/endpoint. More for critical paths. See reference.md for path categories and framework-specific discovery.","EXECUTE: Run every test case in the matrix. Capture real output. Mark PASS/FAIL with evidence. Note anything unexpected. Re-run CRITICAL/MAJOR failures to confirm non-transient.","REPORT: Structured findings with interface surface, test matrix, results by category, findings with severity, tool recommendations for developer follow-up."]}
{"kind":"category","id":"happy_paths","label":"Happy Paths","description":"Valid inputs, standard workflows, expected use cases","examples_cli":["binary --required-arg value","binary subcommand --flag value"],"examples_api":["GET /resource with valid auth","POST /resource with valid JSON payload"]}
{"kind":"category","id":"missing_inputs","label":"Missing Inputs","description":"Required args omitted, empty values, null payloads","examples_cli":["binary (no args)","binary subcommand (missing required arg)","binary --flag ''"],"examples_api":["POST /resource with empty body","POST with missing required field","GET without auth header"]}
{"kind":"category","id":"invalid_inputs","label":"Invalid Inputs","description":"Wrong types, out-of-range, malformed data","examples_cli":["binary --count notanumber","binary --file /nonexistent"],"examples_api":["POST with malformed JSON","GET with invalid ID format"]}
{"kind":"category","id":"boundary_cases","label":"Boundary Cases","description":"Edge of valid ranges, empty, max, zero","examples_cli":["binary --count 0","binary --count 999999999"],"examples_api":["GET ?limit=0","POST with max-length string"]}
{"kind":"category","id":"error_paths","label":"Error Paths","description":"Nonexistent resources, conflicts, permission denied","examples_cli":["binary --config missing.conf","binary subcommand nonexistent"],"examples_api":["GET /resource/99999 (not found)","POST duplicate resource"]}
{"kind":"category","id":"edge_cases","label":"Edge Cases","description":"Special chars, unicode, concurrent requests, signals","examples_cli":["binary --name 'a;b'","binary --name '日本語'","echo data | binary"],"examples_api":["POST with unicode payload","Concurrent identical requests"]}
{"kind":"severity","id":"severity_model","levels":[{"level":"CRITICAL","meaning":"Happy path broken, crash, or panic on valid input","action":"Blocks all use. Developer must fix immediately."},{"level":"MAJOR","meaning":"Error handling broken — wrong exit code, panic on bad input, unhelpful error, leaky errors","action":"Should fix before release."},{"level":"MINOR","meaning":"Cosmetic or UX issue — misleading output, inconsistent formatting, missing --help detail","action":"Fix when convenient."},{"level":"OBSERVATION","meaning":"Works correctly but notable behavior worth documenting or a tool recommendation for the developer","action":"Informational. May include tool suggestions for further investigation."}]}
{"kind":"output","id":"report_format","sections":["## Target — Binary/API path, version, interface type, how it was built","## Interface Surface — Complete list of flags, subcommands, endpoints, HTTP methods discovered","## Test Matrix — Table: ID | Category | Command/Request | Expected | Actual | Status","## Findings — Each: Severity | Category | Description | Reproduction | Evidence (verbatim output)","## Developer Toolbox — Tools and commands the developer can use to investigate further (see tools.md)","## Summary — Total tests, PASS/FAIL counts, severity breakdown, recommended next steps"]}
{"kind":"gate","id":"surface_complete","check":"Every flag from --help tested. Every endpoint from routes/spec tested. No gaps in coverage.","failure":"List uncovered interface elements as gaps in report."}
{"kind":"gate","id":"evidence_backed","check":"Every PASS and FAIL has verbatim command output as proof.","failure":"Re-run the test to capture real output before reporting."}
{"kind":"gate","id":"no_assumptions","check":"No test result based on code reading alone — all from actual invocation.","failure":"Mark as UNVERIFIED and execute for real."}
{"kind":"gate","id":"fail_fast_check","check":"If any happy path CRITICAL failure found, report it immediately before continuing with failure path testing.","failure":"Do not spend time testing edge cases when core functionality is broken."}
{"kind":"ref","file":"reference.md","use":"Path taxonomy, framework-specific discovery patterns, shell testing patterns, test matrix templates"}
{"kind":"ref","file":"tools.md","use":"Tool reference with testing recipes: curl, jq, timeout, diff, file, and more"}
{"kind":"ref","file":"checklist.md","use":"Pre-flight, coverage verification, and report completeness checks"}

Hands-On QA — Manual Invocation Testing

Mandatory Verification Gate

Before delivering the final report, execute these checks:

# 1. Re-verify interface surface completeness
<target> --help 2>&1 | head -80

# 2. Re-run CRITICAL/MAJOR failures to confirm non-transient
# (rerun the exact reproduction command from the finding)

# 3. Cross-reference test matrix against discovered surface
# Every flag, subcommand, and endpoint must appear in the matrix

# 4. Verify no hallucinated results
# Every row in the matrix must have verbatim output evidence

Anti-Hallucination Shield

FORBIDDEN:

• Writing output without running the command
• Paraphrasing or summarizing what output "probably was"
• Marking PASS because the code "looks correct"
• Reusing output from a previous run without re-executing
• Claiming a test was run when it was not

REQUIRED for every result:

• Exact command that was executed
• Verbatim stdout and stderr
• Actual exit code (numeric)
• HTTP status code for API calls

Core Loop

DISCOVER → PLAN → EXECUTE → REPORT
    ↑                       |
    └─── re-discover if new flags/endpoints found during testing

If testing reveals undocumented flags, new subcommands, or unexpected endpoints, loop back to DISCOVER and expand the test matrix before continuing.

When to Stop

• Every discovered interface element has at least 1 happy path test
• Every discovered interface element has at least 1 failure path test
• All CRITICAL and MAJOR findings have clear reproduction steps
• No test result is based on assumption or code reading
• Report passes all four verification gates