# Trace > Use when verifying a repository against its authoritative spec — spec-to-code audits, implementation-completeness reviews, release verification, behavior inventories, repo-wide acceptance testing, or when asked whether every documented requirement is implemented and actually works. - Skill: `2389-research/trace` (Agent Skill, multi-file: 4 files) - Install (CLI): `npx skillmds@latest add 2389-research/trace` - Raw SKILL.md: https://api.skillmd.com/api/skills/2389-research/trace/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: 2389-research (https://skillmd.com/u/2389-research) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/2389-research/trace --- # TRACE — Test Requirements Against Code & Execution ## Objective Maintain exactly one canonical UTF-8 CSV named `behavior_verification.csv` at the repository root. It is the source of truth for every required behavior and every externally observable repository capability, including implementation evidence, executable verification, defects, fixes, and current status. A TRACE run succeeds only when the CSV and the executed repository behavior agree. ## Required outputs - `behavior_verification.csv` — the only canonical verification matrix and the only required mutable record. - `.scratch/` — gitignored disposable scripts, test data, logs, captures, and evidence generated during the run. - `scenarios.jsonl` — reusable scenarios promoted only after they pass. Create it when at least one reusable scenario exists. Do not create a second verification CSV, a renamed copy, a backup CSV, or per-agent CSV fragments anywhere in the repository. TRACE performs no version-control writes. Do not commit, branch, or push unless the operator explicitly instructs it; list every created and modified file in the final report instead. ## Non-negotiable invariants 1. **The main thread is the sole CSV writer.** Reviewers, subagents, and parallel workers may inspect the repository and return candidate findings, but they must never create, edit, normalize, sort, or rewrite `behavior_verification.csv`. 2. **Preserve stable row IDs.** Use `TRACE-0001`, `TRACE-0002`, and so on. Never renumber, recycle, or repurpose an existing ID. Assign each new ID as one greater than the highest ID ever used, even after a row was removed. 3. **One row means one atomic observable behavior.** Split compound requirements when each clause can pass or fail independently. 4. **Do not guess.** Record ambiguity, conflicting sources, missing code, missing credentials, environment blockers, product decisions, and untestable behavior explicitly. 5. **Execution is the verification authority.** Static reading supports discovery and test design but does not verify a required behavior. A required spec item without a practical execution path fails the spec-completion gate. 6. **Keep required intent distinct from current implementation.** The authoritative spec defines what is required. Executable code defines what currently exists. When they conflict, record both, use code as the authority for current behavior, and log the spec/code mismatch rather than silently changing the expected behavior. 7. **Use real dependencies where practical.** Prefer local real services, disposable data, test-mode or sandbox external services, and real authentication flows. Do not replace the behavior under test with mocks. 8. **Do not mutate product behavior while testing.** Testing may create disposable fixtures, configuration, test accounts, or local services. Product changes belong only in the Fix step and must be tied to logged defects. 9. **Never expose or persist secrets.** Record environment-variable names, credential requirements, and redacted identifiers—not secret values, tokens, session cookies, or private keys. 10. **`Verified` is earned, not inferred.** A row becomes `Verified` only after a passing execution, any required fix and re-test, a fresh-eyes review, and a CSV evidence audit. ## Canonical CSV schema The header must match this exact order and spelling: ```csv ID,Area,Capability,Actor,Story,Expected behavior from spec/code,Interface / entrypoint,Source,Test method,Status,Defects,Defect type,Actual result,Notes / open questions ``` Allowed `Status` values: - `Spec'd` — discovered and mapped, but not yet successfully executed and reviewed. - `Tested-Pass` — execution passed; fresh-eyes review and CSV audit are still pending. - `Tested-Fail` — execution failed or observed behavior contradicted the expected behavior. - `Fixed` — a scoped fix was applied; passing re-test is still pending. - `Verified` — execution passed after any fix, review found no unresolved issue, and the CSV matches the evidence. Allowed `Defect type` values: - `Functional` - `Logistical` - `UX` - `Documentation` - `Testability` - `Environment` - `Unknown` Leave `Defect type` blank when there is no defect. Do not invent additional status or defect values. ### Field contract | Column | Required content | |---|---| | `ID` | Stable `TRACE-NNNN` identifier. | | `Area` | Product or system area, such as Auth, CLI, API, Persistence, Worker, Export, or Build. | | `Capability` | Concise name for one externally observable behavior. | | `Actor` | Human role, external system, operator, developer, scheduler, or other initiator. | | `Story` | Actor-centered scenario, normally "As …, when …, I can …". | | `Expected behavior from spec/code` | Atomic, observable pass condition. Preserve spec intent and note code behavior when it differs. | | `Interface / entrypoint` | Public route, command, binary, function export, queue, event, UI path, config surface, or other callable boundary. | | `Source` | Precise evidence such as `spec:path:lines`, `code:path:symbol`, test name, manifest, route registration, or `code-only`. Include both sides of a mismatch. | | `Test method` | Strongest practical method, dependencies, setup, invocation, assertions, and cleanup. | | `Status` | One allowed lifecycle value. | | `Defects` | Concrete observed defect, mismatch, blocker, or empty. Avoid vague phrases such as "doesn't work." | | `Defect type` | One allowed classification or empty. | | `Actual result` | Executed evidence: timestamp, command or manual procedure, exit code, observed result, concise output/log summary, and evidence path when useful. | | `Notes / open questions` | Ambiguity, scope disposition, attempt count, credentials, decisions, cleanup caveats, or next action. | Use one-line cells where practical. Quote cells correctly when commas, quotes, or newlines are unavoidable. Preserve UTF-8 without a byte-order mark. ### Execution evidence format Use a compact, factual structure in `Actual result`: ```text 2026-08-18T15:04:05Z | command: | exit: | observed: | output: | evidence: ``` Do not paste enormous logs into the CSV. Put full disposable logs under `.scratch/evidence//` and retain the exact command and decisive output summary in the row. Use the actual execution time in each timestamp (for example from `date -u +%Y-%m-%dT%H:%M:%SZ`); never a placeholder or a rounded guess. When a row is re-tested, append a new entry; never overwrite prior failing evidence. ## Status transitions Normal no-fix path: ```text Spec'd -> Tested-Pass -> Verified ``` Normal defect path: ```text Spec'd -> Tested-Fail -> Fixed -> Tested-Pass -> Verified ``` A failed re-test returns the row to `Tested-Fail`. A fresh-eyes finding also returns the affected row to `Tested-Fail` unless a fix has already been applied, in which case use `Fixed` until re-tested. Never change a failed row directly to `Verified` based on inspection, confidence, or a code diff. The order is fixed: passing execution, then fresh-eyes review, then the fresh-context CSV audit. Only the audit promotes `Tested-Pass` to `Verified`. Write each transition to the CSV as it occurs; never batch transitions. A fixed-but-not-yet-re-tested row must read `Fixed` in the CSV, not be held back until it can be written as `Tested-Pass`. ## Startup procedure 1. Locate the repository root. 2. Search the repository for every file named `behavior_verification.csv`. - If none exists, create one at the root with the exact header. - If exactly one exists, use it in place and preserve all IDs. - If more than one exists, stop mutation, report the duplicate paths, and resolve which one is canonical before continuing. Do not merge by guessing. 3. Read the existing CSV before assigning IDs. The next ID is one greater than the highest existing numeric suffix. 4. Ensure `/.scratch/` or `.scratch/` is present in the repository root `.gitignore`. 5. Inspect any existing `scenarios.jsonl`; do not overwrite passing scenarios. 6. Identify the user-designated or repository-designated authoritative spec. If multiple candidates conflict and precedence is not explicit, record the ambiguity and treat it as a gate blocker. 7. Run the bundled read-only validator after initial setup and after each write batch: ```bash python3 /scripts/trace_csv.py validate --repo . ``` The main thread may use the helper to initialize the CSV. Reviewers must use only read-only commands. ## Phase 0 — Project-shape discovery Build a factual map before extracting requirements or designing scenarios. ### Inventory Detect and record: - languages and runtime versions; - frameworks and generated-code systems; - package managers, lockfiles, build tools, task runners, and test tools; - executable entrypoints, binaries, scripts, CLIs, subcommands, and flags; - public APIs, RPC methods, web routes, webhooks, sockets, and exported library surfaces; - user-visible pages, forms, navigation paths, and browser workflows; - workers, queues, schedulers, cron jobs, event consumers, and background processes; - configuration files, environment variables, feature flags, and state directories; - authentication, authorization, roles, tenancy, session handling, and credential boundaries; - persistence engines, schemas, migrations, caches, files, and cleanup behavior; - third-party integrations and their local, sandbox, emulator, or test-mode options; - fixtures, seeds, factories, sample data, and disposable-account mechanisms; - local services, containers, devcontainers, compose files, and service health checks; - CI workflows, release jobs, packaging, generated artifacts, and deployment checks; - existing unit, integration, contract, scenario, system, browser, and end-to-end tests. Use manifests, route registration, command registration, export declarations, generated API schemas, configuration loading, and runtime introspection—not filenames alone. ### Spec authority Identify the authoritative specification and record its path or source. Use this precedence unless the user specifies another: 1. Explicitly user-designated specification or acceptance contract. 2. Repository file explicitly labeled canonical, normative, requirements, protocol, or acceptance criteria. 3. Versioned API/schema/protocol definition used to generate or validate implementation. 4. Product documentation and examples. 5. Tests as evidence of intended behavior. Executable code wins over secondary documentation when determining current behavior. It does not erase an unmet authoritative requirement. ### Discovery strategy Before Phase 1, state internally how capability coverage will be obtained. Use at least these independent passes: 1. **Spec pass:** extract every normative statement and acceptance condition. 2. **Surface pass:** enumerate externally callable routes, commands, exports, UI workflows, workers, events, and configuration effects from code. 3. **Runtime pass:** start or interrogate the system where practical to discover generated routes, command help, schemas, health endpoints, plugin registries, or dynamic capabilities. 4. **Test pass:** inspect existing tests and fixtures for behavior not obvious from registration code. 5. **Independent coverage pass:** review the discovered surface without relying on the current CSV row order. Decide how each dependency will run locally or in a sandbox. Record missing services, credentials, destructive risks, or unavailable platforms immediately; do not defer them until testing. ## Phase 1 — Atomic behavior matrix ### Extract spec requirements For every normative requirement: 1. Identify the actor and trigger. 2. Identify one observable outcome. 3. Split independent clauses into separate rows. 4. Capture exact spec evidence in `Source`. 5. Locate implementation evidence. If none exists, say so explicitly. 6. Identify the actual interface or entrypoint. 7. Design a practical executable test path with setup, assertions, and cleanup. 8. Add or update the row as `Spec'd`. Normative cues include "must," "shall," "should" when acceptance depends on it, explicit acceptance criteria, error behavior, authorization rules, data persistence rules, limits, defaults, and compatibility guarantees. ### Discover code-only capabilities Enumerate every externally observable capability exposed by the repository, even when absent from the authoritative spec. Add one row per atomic capability with `Source` beginning with `code-only` and note one of these dispositions: ```text Scope disposition: verify ``` or ```text Scope disposition: out-of-scope/code-only — ``` An out-of-scope disposition must be explicit and reasoned. It cannot conceal an open `Functional`, `Logistical`, or `UX` defect. An out-of-scope row keeps its current status — normally `Spec'd` — and is exempted from the exit criteria rather than verified. Typical code-only surfaces include undocumented routes, flags, environment-controlled behavior, importable exports, migration commands, maintenance jobs, debug endpoints, compatibility aliases, and error modes. ### Map mismatches without guessing When spec and code disagree: - keep the spec requirement in `Expected behavior from spec/code`; - describe current code behavior separately in the same cell or `Notes / open questions`; - cite both sources; - classify missing or contradictory implementation as `Functional` unless the discrepancy is purely documentation; - record a product-decision blocker when the intended behavior cannot be determined; - do not rewrite the spec expectation to make the implementation pass. ### Fresh-context coverage review After the first matrix is built, perform an independent review against the spec and repository surface. Prefer a fresh subagent or reviewer when available. It must be read-only and return a structured list containing proposed row, evidence, and reason. The main thread decides and writes. When no independent context is available, perform a second pass from manifests, route/command/export registration, generated schemas, and runtime help — or, when the repository lacks such artifacts, an independent re-reading of its public entrypoints — without reading rows sequentially. Compare the resulting capability list to the CSV, then add omissions and remove or qualify unsupported claims. ## Spec-completion gate Do not begin scenario testing until every required spec row has all of the following: - authoritative spec evidence; - implementation evidence that plausibly reaches the expected behavior; - a concrete interface or entrypoint; - a practical execution-based test path; - known setup and cleanup requirements; - no unresolved product decision; - no missing user input, credential, secret, service, platform, or environment prerequisite. A required row that fails this gate stays `Spec'd`. Record the missing evidence or blocker and report it immediately with a table: ```text CSV ID | Requirement | Missing evidence or implementation | Blocker | Next action ``` Examples of gate failures: - required behavior has no implementation; - implementation cannot be identified; - spec ambiguity needs a product decision; - only production credentials exist; - no safe or practical execution path exists; - a required external service has no available sandbox and cannot be exercised safely; - required platform or hardware is unavailable. Continue the verification loop for rows that individually pass the gate; a run with any gate-blocked required row cannot report `SUCCESS`. An implementation that exists but misbehaves is not a gate failure — test it and log the defect. Do not use mocks or static inspection to force a required row through the gate. ## Verification loop Enter this loop for rows that pass the spec-completion gate. Gate-blocked required rows stay `Spec'd` and keep the overall run from reporting `SUCCESS`. ### Test hierarchy For each row not `Verified`, use the strongest practical method in this order: 1. real scenario, end-to-end, or system test; 2. CLI through the public binary or command surface; 3. public API or protocol harness; 4. integration test with real local or sandbox dependencies; 5. unit test through the public behavior boundary; 6. targeted executable harness; 7. static analysis only for discovery or an explicitly out-of-scope code-only surface when execution is genuinely impossible. Static analysis never advances a row to `Tested-Pass` or `Verified`. Prefer black-box assertions on externally observable behavior. Use internal inspection only to explain failures or assert state that has no public observation path. ### Scratch assets Put disposable scenario scripts, captures, logs, temporary data, and local orchestration under: ```text .scratch/ ``` Recommended layout: ```text .scratch/ evidence// scripts/ fixtures/ state/ ``` Do not commit scratch artifacts. Ensure cleanup is safe and idempotent. ### 1. Test For every unverified row: 1. Create isolated, disposable state. 2. Start real local dependencies or approved sandbox services. 3. Invoke the documented public interface. 4. Assert the expected success behavior, error behavior, state transition, authorization rule, and cleanup where relevant. 5. Capture the exact command or manual procedure, exit code, decisive output, log summary, and evidence path. 6. Do not change product behavior during this step. 7. Set `Tested-Pass` or `Tested-Fail` and update defect fields factually. A test is not strong merely because it passes. It must be capable of failing when the required behavior is broken. ### 2. Fix For each logged `Functional`, `Logistical`, or `UX` defect: 1. Diagnose the root cause from execution evidence. 2. Make the smallest coherent product change that resolves the logged defect. 3. Avoid unrelated refactors, style churn, dependency upgrades, or speculative improvements. 4. Add or strengthen a regression test at the most appropriate layer. 5. Mark only affected rows `Fixed`. 6. Record changed files and the root cause in `Defects`, `Actual result`, or `Notes / open questions`. Fix `Documentation`, `Testability`, `Environment`, or `Unknown` defects only when the correction is obvious, safe, and needs no product decision. Otherwise record the next action and stop or continue according to the gate and safety rules. ### 3. Re-test Re-run the exact failing scenario and any directly affected regression or neighboring scenario. - Pass: set `Tested-Pass`, append the new execution evidence after the preserved failing evidence in `Actual result`, then perform fresh-eyes review. - Fail: set `Tested-Fail`, preserve the attempt evidence, increment the cycle count, and return to diagnosis if below the safety cap. Never mark `Verified` merely because tests compile, a unit test passes while the public scenario still fails, or the diff appears correct. ### 4. Fresh-eyes review After a passing execution, inspect the touched code, tests, scenario, and CSV row for: - authentication and authorization bypasses; - tenant, role, and ownership isolation; - input validation and output encoding; - injection, traversal, unsafe deserialization, secret leakage, and insecure defaults; - incorrect error codes, error messages, and partial-success behavior; - boundary values, empty values, malformed values, duplicates, retries, and idempotency; - stale state, transactionality, race conditions, ordering, locking, and cleanup; - persistence, migration, cache, and restart behavior; - business-rule contradictions and spec/code drift; - performance traps, unbounded work, resource leaks, and timeout behavior; - flaky assertions, hidden test coupling, and tests that cannot detect regressions; - unsupported or overstated claims in the CSV. Log every material finding as a new row or update the affected row. Fix and re-test scoped `Functional`, `Logistical`, and `UX` findings. A row can advance to `Verified` only when this review is clear. ### 5. Fresh-context CSV audit After each write batch and before final reporting: 1. Validate exact headers, UTF-8, parseability, allowed enums, and unique IDs. 2. Confirm every command, exit code, result, fix, source, and status matches actual evidence. 3. Confirm no row says `Verified` without passing execution evidence. 4. Confirm failed attempts and unresolved defects were not erased. 5. Confirm code-only rows have explicit scope disposition. 6. Confirm no second `behavior_verification.csv` exists. 7. Run: ```bash python3 /scripts/trace_csv.py validate --repo . ``` When the audit passes, change eligible `Tested-Pass` rows to `Verified` and run the validator again. ## Reusable scenarios Promote a passing scratch scenario to root-level `scenarios.jsonl` when it is deterministic, safe, reusable, and exercises a public behavior boundary. Store one JSON object per line. Recommended record shape: ```json {"id":"scenario-trace-0007-happy-path","capability_ids":["TRACE-0007"],"name":"Create and retrieve a record through the public API","preconditions":["service running","TEST_DATABASE_URL set"],"command":"python3 scenarios/create_and_get.py","expected":{"exit_code":0,"assertions":["created id is returned","retrieved payload matches"]},"cleanup":"scenario deletes its disposable record","last_verified":"2026-08-18T15:04:05Z"} ``` Rules: - Reference one or more CSV IDs in `capability_ids`. - Store environment-variable names, never values. - Write commands to run from the repository root. - Store a promoted scenario's script at a committed path such as `scenarios/`; a promoted scenario must never reference `.scratch/`. - Include deterministic assertions and cleanup. - Do not promote failing, flaky, destructive, environment-specific, or one-off diagnostic scripts. - Preserve existing scenario IDs and append new records rather than silently replacing history. ## Safety cap Track failed test/fix/re-test cycles per capability in `Notes / open questions`: ```text Cycle 1/3: ... Cycle 2/3: ... Cycle 3/3: ... ``` After three failed cycles for one capability: 1. Stop modifying that capability. 2. Leave it `Tested-Fail`. 3. Preserve all three attempt summaries, root-cause analysis, commands, evidence paths, and changed files. 4. Record the most specific known next action. 5. Continue independent capabilities when safe, but the overall run cannot report success. 6. Include the capped capability prominently in the final report. Do not reset the attempt count by renaming the capability, creating a new row, or slightly changing the test. ## Successful exit criteria Report success only when all conditions are true: - every required spec capability is `Verified`; - every exposed code capability is `Verified` or explicitly marked `Scope disposition: out-of-scope/code-only` with no open `Functional`, `Logistical`, or `UX` defect; - no open `Functional`, `Logistical`, or `UX` defects remain; - every reusable passing scenario is represented in `scenarios.jsonl`; - the final fresh-eyes review has no unresolved material finding; - the final fresh-context CSV audit passes; - exactly one canonical UTF-8 `behavior_verification.csv` exists; - the CSV matches the commands and observed evidence actually produced. ## Final response contract Return: 1. Overall result: `SUCCESS`, `BLOCKED AT SPEC-COMPLETION GATE`, or `INCOMPLETE — TEST FAILURES REMAIN`. When gate blockers and test failures both remain, report `BLOCKED AT SPEC-COMPLETION GATE` and include the failures. 2. CSV path and row counts by status. 3. Required-spec coverage and code-only coverage. 4. Tests executed, grouped by interface and method. 5. Defects found, root causes, fixes, and re-test outcomes. 6. Remaining blockers or safety-capped failures with CSV IDs and next actions. 7. Promoted `scenarios.jsonl` entries. 8. Final validator result and confirmation that exactly one canonical CSV exists. Do not claim complete verification when any success condition is unmet. ## Anti-patterns Never: - write the CSV from multiple agents or processes; - renumber IDs after sorting or inserting rows; - treat documentation prose as proof of runtime behavior; - use mocks when the real local or sandbox dependency is practical; - weaken a test merely to make it pass; - alter expected behavior to match a bug; - fix unlogged or unrelated product behavior during the loop; - hide blockers in vague notes; - mark static inspection as execution; - mark `Verified` before re-test and review; - erase failed evidence after a fix; - store credentials or personal data in evidence; - touch production data or services without explicit authorization; - declare success with an open required row or an open Functional, Logistical, or UX defect.