Trace

TRACE — Test Requirements Against Code & Execution

Objective

Maintain exactly one canonical UTF-8 CSV named behavior_verification.csv at the repository root. It is the source of truth for every required behavior and every externally observable repository capability, including implementation evidence, executable verification, defects, fixes, and current status.

A TRACE run succeeds only when the CSV and the executed repository behavior agree.

Required outputs

• behavior_verification.csv — the only canonical verification matrix and the only required mutable record.
• .scratch/ — gitignored disposable scripts, test data, logs, captures, and evidence generated during the run.
• scenarios.jsonl — reusable scenarios promoted only after they pass. Create it when at least one reusable scenario exists.

Do not create a second verification CSV, a renamed copy, a backup CSV, or per-agent CSV fragments anywhere in the repository.

TRACE performs no version-control writes. Do not commit, branch, or push unless the operator explicitly instructs it; list every created and modified file in the final report instead.

Non-negotiable invariants

1. The main thread is the sole CSV writer. Reviewers, subagents, and parallel workers may inspect the repository and return candidate findings, but they must never create, edit, normalize, sort, or rewrite behavior_verification.csv.
2. Preserve stable row IDs. Use TRACE-0001, TRACE-0002, and so on. Never renumber, recycle, or repurpose an existing ID. Assign each new ID as one greater than the highest ID ever used, even after a row was removed.
3. One row means one atomic observable behavior. Split compound requirements when each clause can pass or fail independently.
4. Do not guess. Record ambiguity, conflicting sources, missing code, missing credentials, environment blockers, product decisions, and untestable behavior explicitly.
5. Execution is the verification authority. Static reading supports discovery and test design but does not verify a required behavior. A required spec item without a practical execution path fails the spec-completion gate.
6. Keep required intent distinct from current implementation. The authoritative spec defines what is required. Executable code defines what currently exists. When they conflict, record both, use code as the authority for current behavior, and log the spec/code mismatch rather than silently changing the expected behavior.
7. Use real dependencies where practical. Prefer local real services, disposable data, test-mode or sandbox external services, and real authentication flows. Do not replace the behavior under test with mocks.
8. Do not mutate product behavior while testing. Testing may create disposable fixtures, configuration, test accounts, or local services. Product changes belong only in the Fix step and must be tied to logged defects.
9. Never expose or persist secrets. Record environment-variable names, credential requirements, and redacted identifiers—not secret values, tokens, session cookies, or private keys.
10. Verified is earned, not inferred. A row becomes Verified only after a passing execution, any required fix and re-test, a fresh-eyes review, and a CSV evidence audit.

Canonical CSV schema

The header must match this exact order and spelling:

ID,Area,Capability,Actor,Story,Expected behavior from spec/code,Interface / entrypoint,Source,Test method,Status,Defects,Defect type,Actual result,Notes / open questions

Allowed Status values:

• Spec'd — discovered and mapped, but not yet successfully executed and reviewed.
• Tested-Pass — execution passed; fresh-eyes review and CSV audit are still pending.
• Tested-Fail — execution failed or observed behavior contradicted the expected behavior.
• Fixed — a scoped fix was applied; passing re-test is still pending.
• Verified — execution passed after any fix, review found no unresolved issue, and the CSV matches the evidence.

Allowed Defect type values:

• Functional
• Logistical
• UX
• Documentation
• Testability
• Environment
• Unknown

Leave Defect type blank when there is no defect. Do not invent additional status or defect values.

Field contract

Use one-line cells where practical. Quote cells correctly when commas, quotes, or newlines are unavoidable. Preserve UTF-8 without a byte-order mark.

Execution evidence format

Use a compact, factual structure in Actual result:

2026-08-18T15:04:05Z | command: <exact command or manual procedure> | exit: <code or n/a> | observed: <what happened> | output: <key stdout/stderr/log summary> | evidence: <optional .scratch path>

Do not paste enormous logs into the CSV. Put full disposable logs under .scratch/evidence/<ID>/ and retain the exact command and decisive output summary in the row.

Use the actual execution time in each timestamp (for example from date -u +%Y-%m-%dT%H:%M:%SZ); never a placeholder or a rounded guess. When a row is re-tested, append a new entry; never overwrite prior failing evidence.

Status transitions

Normal no-fix path:

Spec'd -> Tested-Pass -> Verified

Normal defect path:

Spec'd -> Tested-Fail -> Fixed -> Tested-Pass -> Verified

A failed re-test returns the row to Tested-Fail. A fresh-eyes finding also returns the affected row to Tested-Fail unless a fix has already been applied, in which case use Fixed until re-tested.

Never change a failed row directly to Verified based on inspection, confidence, or a code diff.

The order is fixed: passing execution, then fresh-eyes review, then the fresh-context CSV audit. Only the audit promotes Tested-Pass to Verified.

Write each transition to the CSV as it occurs; never batch transitions. A fixed-but-not-yet-re-tested row must read Fixed in the CSV, not be held back until it can be written as Tested-Pass.

Startup procedure

1. Locate the repository root.
2. Search the repository for every file named behavior_verification.csv.
   • If none exists, create one at the root with the exact header.
   • If exactly one exists, use it in place and preserve all IDs.
   • If more than one exists, stop mutation, report the duplicate paths, and resolve which one is canonical before continuing. Do not merge by guessing.
3. Read the existing CSV before assigning IDs. The next ID is one greater than the highest existing numeric suffix.
4. Ensure /.scratch/ or .scratch/ is present in the repository root .gitignore.
5. Inspect any existing scenarios.jsonl; do not overwrite passing scenarios.
6. Identify the user-designated or repository-designated authoritative spec. If multiple candidates conflict and precedence is not explicit, record the ambiguity and treat it as a gate blocker.
7. Run the bundled read-only validator after initial setup and after each write batch:

python3 <skill-directory>/scripts/trace_csv.py validate --repo .

The main thread may use the helper to initialize the CSV. Reviewers must use only read-only commands.

Phase 0 — Project-shape discovery

Build a factual map before extracting requirements or designing scenarios.

Inventory

Detect and record:

• languages and runtime versions;
• frameworks and generated-code systems;
• package managers, lockfiles, build tools, task runners, and test tools;
• executable entrypoints, binaries, scripts, CLIs, subcommands, and flags;
• public APIs, RPC methods, web routes, webhooks, sockets, and exported library surfaces;
• user-visible pages, forms, navigation paths, and browser workflows;
• workers, queues, schedulers, cron jobs, event consumers, and background processes;
• configuration files, environment variables, feature flags, and state directories;
• authentication, authorization, roles, tenancy, session handling, and credential boundaries;
• persistence engines, schemas, migrations, caches, files, and cleanup behavior;
• third-party integrations and their local, sandbox, emulator, or test-mode options;
• fixtures, seeds, factories, sample data, and disposable-account mechanisms;
• local services, containers, devcontainers, compose files, and service health checks;
• CI workflows, release jobs, packaging, generated artifacts, and deployment checks;
• existing unit, integration, contract, scenario, system, browser, and end-to-end tests.

Use manifests, route registration, command registration, export declarations, generated API schemas, configuration loading, and runtime introspection—not filenames alone.

Spec authority

Identify the authoritative specification and record its path or source. Use this precedence unless the user specifies another:

1. Explicitly user-designated specification or acceptance contract.
2. Repository file explicitly labeled canonical, normative, requirements, protocol, or acceptance criteria.
3. Versioned API/schema/protocol definition used to generate or validate implementation.
4. Product documentation and examples.
5. Tests as evidence of intended behavior.

Executable code wins over secondary documentation when determining current behavior. It does not erase an unmet authoritative requirement.

Discovery strategy

Before Phase 1, state internally how capability coverage will be obtained. Use at least these independent passes:

1. Spec pass: extract every normative statement and acceptance condition.
2. Surface pass: enumerate externally callable routes, commands, exports, UI workflows, workers, events, and configuration effects from code.
3. Runtime pass: start or interrogate the system where practical to discover generated routes, command help, schemas, health endpoints, plugin registries, or dynamic capabilities.
4. Test pass: inspect existing tests and fixtures for behavior not obvious from registration code.
5. Independent coverage pass: review the discovered surface without relying on the current CSV row order.

Decide how each dependency will run locally or in a sandbox. Record missing services, credentials, destructive risks, or unavailable platforms immediately; do not defer them until testing.

Phase 1 — Atomic behavior matrix

Extract spec requirements

For every normative requirement:

1. Identify the actor and trigger.
2. Identify one observable outcome.
3. Split independent clauses into separate rows.
4. Capture exact spec evidence in Source.
5. Locate implementation evidence. If none exists, say so explicitly.
6. Identify the actual interface or entrypoint.
7. Design a practical executable test path with setup, assertions, and cleanup.
8. Add or update the row as Spec'd.

Normative cues include "must," "shall," "should" when acceptance depends on it, explicit acceptance criteria, error behavior, authorization rules, data persistence rules, limits, defaults, and compatibility guarantees.

Discover code-only capabilities

Enumerate every externally observable capability exposed by the repository, even when absent from the authoritative spec. Add one row per atomic capability with Source beginning with code-only and note one of these dispositions:

Scope disposition: verify

or

Scope disposition: out-of-scope/code-only — <reason>

An out-of-scope disposition must be explicit and reasoned. It cannot conceal an open Functional, Logistical, or UX defect. An out-of-scope row keeps its current status — normally Spec'd — and is exempted from the exit criteria rather than verified.

Typical code-only surfaces include undocumented routes, flags, environment-controlled behavior, importable exports, migration commands, maintenance jobs, debug endpoints, compatibility aliases, and error modes.

Map mismatches without guessing

When spec and code disagree:

• keep the spec requirement in Expected behavior from spec/code;
• describe current code behavior separately in the same cell or Notes / open questions;
• cite both sources;
• classify missing or contradictory implementation as Functional unless the discrepancy is purely documentation;
• record a product-decision blocker when the intended behavior cannot be determined;
• do not rewrite the spec expectation to make the implementation pass.

Fresh-context coverage review

After the first matrix is built, perform an independent review against the spec and repository surface.

Prefer a fresh subagent or reviewer when available. It must be read-only and return a structured list containing proposed row, evidence, and reason. The main thread decides and writes.

When no independent context is available, perform a second pass from manifests, route/command/export registration, generated schemas, and runtime help — or, when the repository lacks such artifacts, an independent re-reading of its public entrypoints — without reading rows sequentially. Compare the resulting capability list to the CSV, then add omissions and remove or qualify unsupported claims.

Spec-completion gate

Do not begin scenario testing until every required spec row has all of the following:

• authoritative spec evidence;
• implementation evidence that plausibly reaches the expected behavior;
• a concrete interface or entrypoint;
• a practical execution-based test path;
• known setup and cleanup requirements;
• no unresolved product decision;
• no missing user input, credential, secret, service, platform, or environment prerequisite.

A required row that fails this gate stays Spec'd. Record the missing evidence or blocker and report it immediately with a table:

CSV ID | Requirement | Missing evidence or implementation | Blocker | Next action

Examples of gate failures:

• required behavior has no implementation;
• implementation cannot be identified;
• spec ambiguity needs a product decision;
• only production credentials exist;
• no safe or practical execution path exists;
• a required external service has no available sandbox and cannot be exercised safely;
• required platform or hardware is unavailable.

Continue the verification loop for rows that individually pass the gate; a run with any gate-blocked required row cannot report SUCCESS. An implementation that exists but misbehaves is not a gate failure — test it and log the defect.

Do not use mocks or static inspection to force a required row through the gate.

Verification loop

Enter this loop for rows that pass the spec-completion gate. Gate-blocked required rows stay Spec'd and keep the overall run from reporting SUCCESS.

Test hierarchy

For each row not Verified, use the strongest practical method in this order:

1. real scenario, end-to-end, or system test;
2. CLI through the public binary or command surface;
3. public API or protocol harness;
4. integration test with real local or sandbox dependencies;
5. unit test through the public behavior boundary;
6. targeted executable harness;
7. static analysis only for discovery or an explicitly out-of-scope code-only surface when execution is genuinely impossible.

Static analysis never advances a row to Tested-Pass or Verified.

Prefer black-box assertions on externally observable behavior. Use internal inspection only to explain failures or assert state that has no public observation path.

Scratch assets

Put disposable scenario scripts, captures, logs, temporary data, and local orchestration under:

.scratch/

Recommended layout:

.scratch/
  evidence/<TRACE-ID>/
  scripts/
  fixtures/
  state/

Do not commit scratch artifacts. Ensure cleanup is safe and idempotent.

1. Test

For every unverified row:

1. Create isolated, disposable state.
2. Start real local dependencies or approved sandbox services.
3. Invoke the documented public interface.
4. Assert the expected success behavior, error behavior, state transition, authorization rule, and cleanup where relevant.
5. Capture the exact command or manual procedure, exit code, decisive output, log summary, and evidence path.
6. Do not change product behavior during this step.
7. Set Tested-Pass or Tested-Fail and update defect fields factually.

A test is not strong merely because it passes. It must be capable of failing when the required behavior is broken.

2. Fix

For each logged Functional, Logistical, or UX defect:

1. Diagnose the root cause from execution evidence.
2. Make the smallest coherent product change that resolves the logged defect.
3. Avoid unrelated refactors, style churn, dependency upgrades, or speculative improvements.
4. Add or strengthen a regression test at the most appropriate layer.
5. Mark only affected rows Fixed.
6. Record changed files and the root cause in Defects, Actual result, or Notes / open questions.

Fix Documentation, Testability, Environment, or Unknown defects only when the correction is obvious, safe, and needs no product decision. Otherwise record the next action and stop or continue according to the gate and safety rules.

3. Re-test

Re-run the exact failing scenario and any directly affected regression or neighboring scenario.

• Pass: set Tested-Pass, append the new execution evidence after the preserved failing evidence in Actual result, then perform fresh-eyes review.
• Fail: set Tested-Fail, preserve the attempt evidence, increment the cycle count, and return to diagnosis if below the safety cap.

Never mark Verified merely because tests compile, a unit test passes while the public scenario still fails, or the diff appears correct.

4. Fresh-eyes review

After a passing execution, inspect the touched code, tests, scenario, and CSV row for:

• authentication and authorization bypasses;
• tenant, role, and ownership isolation;
• input validation and output encoding;
• injection, traversal, unsafe deserialization, secret leakage, and insecure defaults;
• incorrect error codes, error messages, and partial-success behavior;
• boundary values, empty values, malformed values, duplicates, retries, and idempotency;
• stale state, transactionality, race conditions, ordering, locking, and cleanup;
• persistence, migration, cache, and restart behavior;
• business-rule contradictions and spec/code drift;
• performance traps, unbounded work, resource leaks, and timeout behavior;
• flaky assertions, hidden test coupling, and tests that cannot detect regressions;
• unsupported or overstated claims in the CSV.

Log every material finding as a new row or update the affected row. Fix and re-test scoped Functional, Logistical, and UX findings. A row can advance to Verified only when this review is clear.

5. Fresh-context CSV audit

After each write batch and before final reporting:

1. Validate exact headers, UTF-8, parseability, allowed enums, and unique IDs.
2. Confirm every command, exit code, result, fix, source, and status matches actual evidence.
3. Confirm no row says Verified without passing execution evidence.
4. Confirm failed attempts and unresolved defects were not erased.
5. Confirm code-only rows have explicit scope disposition.
6. Confirm no second behavior_verification.csv exists.
7. Run:

python3 <skill-directory>/scripts/trace_csv.py validate --repo .

When the audit passes, change eligible Tested-Pass rows to Verified and run the validator again.

Reusable scenarios

Promote a passing scratch scenario to root-level scenarios.jsonl when it is deterministic, safe, reusable, and exercises a public behavior boundary. Store one JSON object per line.

Recommended record shape:

{"id":"scenario-trace-0007-happy-path","capability_ids":["TRACE-0007"],"name":"Create and retrieve a record through the public API","preconditions":["service running","TEST_DATABASE_URL set"],"command":"python3 scenarios/create_and_get.py","expected":{"exit_code":0,"assertions":["created id is returned","retrieved payload matches"]},"cleanup":"scenario deletes its disposable record","last_verified":"2026-08-18T15:04:05Z"}

Rules:

• Reference one or more CSV IDs in capability_ids.
• Store environment-variable names, never values.
• Write commands to run from the repository root.
• Store a promoted scenario's script at a committed path such as scenarios/; a promoted scenario must never reference .scratch/.
• Include deterministic assertions and cleanup.
• Do not promote failing, flaky, destructive, environment-specific, or one-off diagnostic scripts.
• Preserve existing scenario IDs and append new records rather than silently replacing history.

Safety cap

Track failed test/fix/re-test cycles per capability in Notes / open questions:

Cycle 1/3: ...
Cycle 2/3: ...
Cycle 3/3: ...

After three failed cycles for one capability:

1. Stop modifying that capability.
2. Leave it Tested-Fail.
3. Preserve all three attempt summaries, root-cause analysis, commands, evidence paths, and changed files.
4. Record the most specific known next action.
5. Continue independent capabilities when safe, but the overall run cannot report success.
6. Include the capped capability prominently in the final report.

Do not reset the attempt count by renaming the capability, creating a new row, or slightly changing the test.

Successful exit criteria

Report success only when all conditions are true:

• every required spec capability is Verified;
• every exposed code capability is Verified or explicitly marked Scope disposition: out-of-scope/code-only with no open Functional, Logistical, or UX defect;
• no open Functional, Logistical, or UX defects remain;
• every reusable passing scenario is represented in scenarios.jsonl;
• the final fresh-eyes review has no unresolved material finding;
• the final fresh-context CSV audit passes;
• exactly one canonical UTF-8 behavior_verification.csv exists;
• the CSV matches the commands and observed evidence actually produced.

Final response contract

Return:

1. Overall result: SUCCESS, BLOCKED AT SPEC-COMPLETION GATE, or INCOMPLETE — TEST FAILURES REMAIN. When gate blockers and test failures both remain, report BLOCKED AT SPEC-COMPLETION GATE and include the failures.
2. CSV path and row counts by status.
3. Required-spec coverage and code-only coverage.
4. Tests executed, grouped by interface and method.
5. Defects found, root causes, fixes, and re-test outcomes.
6. Remaining blockers or safety-capped failures with CSV IDs and next actions.
7. Promoted scenarios.jsonl entries.
8. Final validator result and confirmation that exactly one canonical CSV exists.

Do not claim complete verification when any success condition is unmet.

Anti-patterns

Never:

• write the CSV from multiple agents or processes;
• renumber IDs after sorting or inserting rows;
• treat documentation prose as proof of runtime behavior;
• use mocks when the real local or sandbox dependency is practical;
• weaken a test merely to make it pass;
• alter expected behavior to match a bug;
• fix unlogged or unrelated product behavior during the loop;
• hide blockers in vague notes;
• mark static inspection as execution;
• mark Verified before re-test and review;
• erase failed evidence after a fix;
• store credentials or personal data in evidence;
• touch production data or services without explicit authorization;
• declare success with an open required row or an open Functional, Logistical, or UX defect.