Live Test Matrix

live-test-matrix

This skill captures a battle-tested workflow for comprehensively testing an abadge feature by exercising the running dev stack — real Postgres, real Hono+tRPC API on the wrangler Cloudflare-Worker emulator, real Better Auth bearer-token sessions, real agent legacy-API-key bearer tokens, real CLI binary, and real DB state verification.

The methodology has four parts:

1. Test plan: enumerate every test as a row in TESTING.md, organized into four categories (happy / edge / adversarial / pentest), with at least 3 variations per scenario class.
2. Live stack: bring up the dev stack and bootstrap a real authenticated session.
3. Harness: write a bash script that hits the running API + CLI binary with assertions for each row.
4. Verify: check pass/fail state, audit-log invariants, and tear down cleanly.

Why a matrix and not just a single happy-path test? Because real bugs hide in the corners. The cross-profile AAD-tampering finding from PR #119 (where a DB-write attacker rebinding profile_id correctly fails to decrypt) was only discovered because the matrix forced us to write a sanity test that tripped over the AAD invariant. Single-shot testing finds zero of those.

When to use

Trigger this skill when the user wants any of:

• "pentest this feature", "adversarial test", "security test"
• "manually run the CLI", "actually test it not just code tests"
• "test this thoroughly" (when "thoroughly" implies more than ~3 assertions)
• "happy path, edge cases, and adversarial" (or any subset)
• A test matrix or test plan against a live stack
• End-to-end coverage that exercises the wire format the CLI/web/MCP actually use

Do not use for:

• Pure code-level integration tests that belong in packages/*/src/**/__tests__/ (those are bun test + seedX helpers; no live stack)
• Static gates (typecheck, lint, format) — those run via bun run typecheck etc.
• Trivial smoke checks ("does the API start?") that don't benefit from category structure

The four test categories

Every matrix gets ≥3 variations per category. Three is the floor, not the target — add more when the feature has more dimensions to cover.

H — Happy paths

The flows the feature is built for. Cover the dimensions of input space.

For permissions: local + SM + 1 cap, local + SM + 3-cap batch, remote + SM + reveal_plaintext. Three dimensions: agent locality, storage mode, batch size.

E — Edge cases

Boundary conditions, unusual but valid inputs, multi-X-within-Y combinations, list-filter compositions, expiry edges, re-grant-after-revoke.

For permissions: 3-cap batch with shared expiry, list({agentId, itemId}) AND-combined, re-grant the revoked cap (no PERMISSION_ALREADY_EXISTS).

A — Adversarial

Inputs that should be rejected with structured errors. The point is to verify the rejection path, not to break things.

For permissions: read_ciphertext on SM item (matrix violation), [mount_env, mount_env] (in-input duplicate), [] (empty array).

P — Pentests

Security boundary tests. Always cover at minimum:

• Auth: bogus token → UNAUTHORIZED, no token → UNAUTHORIZED
• Cross-tenant: cross-org agent ID → AGENT_NOT_FOUND, cross-org item ID → ITEM_NOT_FOUND, cross-profile within-org if the feature has profile scope
• Tampering: alter token mid-string, DB-write attacker tampers a column the encryption depends on
• Enumeration: list endpoints exclude things the caller shouldn't see
• Audit completeness: every denied access has a corresponding audit row (the AGENTS.md invariant)

For permissions on PR #119 we ran 10 pentests including AAD-tampering — see references/test-categories.md for the full list.

The workflow

1. Capture the feature under test

Ask the user (or extract from conversation):

• What's the feature? (e.g., "multi-capability permission grants")
• What surfaces are involved? (API / CLI / MCP / Web)
• What's the security boundary? (org / profile / agent / item)
• Are there existing integration tests to draw scenarios from?

Read the feature's primary code (e.g., packages/trpc/src/server/routers/<feature>.ts) and the relevant AGENTS.md invariants. Any "Non-negotiable invariants" line that touches this feature is a hint about pentests to run.

2. Generate the test matrix

Draft TESTING.md from assets/TESTING.md.template. Number tests as <category>.<scenario>.<variation>:

• H.1.1 H.1.2 H.1.3 — happy scenario 1, three variations
• A.2.1 — adversarial scenario 2, variation 1
• P.4.1 — pentest 4, variation 1

Each row: 1-sentence description, expected outcome (success or specific error code), pass criterion (DB row count, error code shape, audit entry).

3. Bring up the live stack

doppler run -- turbo dev --filter='!@abadge/docs' &

Mintlify needs mint which most devs don't have — always exclude @abadge/docs. Wait until both :8787 (API) and :3000 (Web) are listening.

See references/dev-stack.md for env vars, ports, and common errors.

If you can use scripts/start-stack.sh, do — it handles the Doppler invocation and waits for :8787/health to return ok.

4. Bootstrap a real test session

Better Auth requires a verified email. The fast path:

1. Sign up via /api/auth/sign-up/email
2. Direct DB update: UPDATE "user" SET email_verified=true WHERE email=...
3. Sign in via /api/auth/sign-in/email and capture the set-auth-token header (this IS the Bearer token)

scripts/bootstrap-test-user.sh does this in one shot. See also references/auth-bootstrap.md.

5. Run the harness

Use assets/harness.sh.template as a starting point. It includes:

• trpc() and trpc_q() helpers for POST mutations and GET queries
• err_code() for parsing the .error.data.code envelope (NOT .error.json.data.code — see gotchas)
• ok() / fail() / hdr() output helpers and a PASS/FAIL tally at the bottom

For CLI tests, set:

export ABADGE_API_URL=http://localhost:8787
export ABADGE_SESSION_TOKEN="$BEARER_FROM_BOOTSTRAP"

And move ~/.abadge/config.json aside — its apiUrl field overrides ABADGE_API_URL, will silently send your localhost session token to production, and you will spend 20 minutes debugging "Unauthorized" before you find this. See references/cli-conventions.md.

For agent bearer tokens (testing agent-side paths like access.reveal, access.mount, items.listForAgent), create the agent with authMethod: legacy_api_key — the response includes a one-time apiKey field that IS the bearer token.

6. Verify and iterate

After the harness runs:

• Tally PASS/FAIL per category
• For any failure, dig in: was the test wrong, the assertion off, or a real bug?
• Check audit log via psql for invariants — every denied access must have a result='denied' row
• If the harness uncovered a real bug, fix it and re-run. The matrix is fast to re-execute (~10s for 30 scenarios)

7. Save artifacts

• The harness script goes to scripts/<feature>-pentest.sh (executable, committed)
• TESTING.md stays in the repo root (committed). Append the run tally to a "Tally" section.
• If you discover a non-obvious pattern (a gotcha, an invariant assertion, a security property), add it to the relevant references/*.md. The skill compounds over time.

8. Tear down

• Kill background processes: pkill -f "turbo dev" 2>&1; pkill -f "wrangler dev" 2>&1; pkill -f "next dev" 2>&1
• Restore moved configs: mv ~/.abadge/config.json.bak.pentest ~/.abadge/config.json
• Optionally TRUNCATE the test data via psql (the test-user emails are sentinel-prefixed for easy cleanup)

scripts/teardown.sh handles all of this.

Common gotchas (learned the hard way)

These have all cost real time in real runs. Read references/api-conventions.md and references/cli-conventions.md for the full list.

• CLI config priority: ~/.abadge/config.json.apiUrl overrides ABADGE_API_URL. Move config aside before running CLI tests.
• Multi-org user: X-Abadge-Org-Id header is required for sessionProcedure / scopedSessionProcedure once the user has 2+ orgs.
• Items.create input shape: {storageMode, payload} only — no profileId. SM items get profile_id=NULL. Bind via DB UPDATE only if you know the AAD implications (see next bullet).
• AAD binding (SM items): (orgId, profileId, itemId, keyVersion) is bound into the AES-GCM AAD at encrypt. Tampering with profile_id after the fact breaks decrypt with INTERNAL_SERVER_ERROR/500. This is defence-in-depth, not a bug — and worth turning into a pentest.
• Item kinds: enum is fixed (login, api_key, token, json, certificate, ssh_key, opaque). See STANDARD_FIELDS_BY_KIND in packages/core/src/constants.ts. Use token + value field for the simplest test items.
• Rate limit: 100 req/min/org. A full matrix run sits near the edge — insert waits between phases or split harnesses.
• Error envelope: .error.data.code (not .error.json.data.code — that's a different tRPC variant). Domain cause may live at .error.data.cause.code.
• Greptile re-trigger phrase: literal @greptileai review (no extra words). It will +1 react to acknowledge.

Templates and references

Output

When you finish a run, the user has:

1. TESTING.md committed to repo root with H/E/A/P matrix and tally
2. scripts/<feature>-pentest.sh committed and executable
3. A run tally (PASS/FAIL per category) in chat with discovered bugs reported
4. The dev stack stopped, configs restored, no orphaned background processes

The harness is idempotent — anyone can re-run it after the change lands to verify nothing regressed.