Ammalgam Protocol Web3 Audit Response Toolkit Reviewing Audit Reports

Reviewing Web3 Audit Reports

Every finding gets a PoC test. The test determines truth, not the auditor's claims.

This skill orchestrates auditing review sessions. Subagents do the per-finding work — the orchestrator sets up, dispatches, tracks state, and generates the final scorecard.

When to Use

• Reviewing a smart contract audit report against the audited codebase
• Creating PoC tests for smart contract audit findings
• Producing a scorecard grading audit accuracy

When NOT to Use

• Implementing fixes for findings (use resolving-audit-findings)
• Initial security review without an existing report (use differential-review)
• Code quality or gas optimization review

Contents

• Inputs / Scope
• Phase 0: Setup
• Phase 1: Dispatch and Track — batching, STATE.md protocol
• Phase 2: Final Scorecard
• Subagent Prompt — passed verbatim to each agent
• Test Patterns — canonical PoC structures per framework
• Severity Reference
• Common Mistakes / Quality Checklist

Rationalizations (Do Not Skip)

Inputs

Scope

Process all findings in the report. One report per session.

Phase 0: Setup (Subagent)

Phase 0 runs entirely inside a subagent to keep the orchestrator's context empty for Phase 1 dispatch. The orchestrator dispatches one setup agent and receives back only the path to the initialized STATE.md.

Setup Agent Instructions

Dispatch a single general-purpose agent with this task:

You are the setup agent for an audit review session. Perform ALL of the following steps, save outputs to disk, and return a 1-line summary.

1. Checkout audited commit: git stash && git checkout {commit_hash}

2. Read the full audit report at {report_path}

3. Discover the testing stack — detect framework, existing test patterns, shared fixtures, build commands. Dispatch up to 2 Explore agents in parallel: one for source structure + contract architecture, one for test stack + existing fixtures.

4. Select the test pattern file and PoC filename — match the project's framework:

   • Foundry/Forge → test_patterns/foundry.sol, PoC filename: POC.sol
   • Hardhat → test_patterns/hardhat.ts, PoC filename: POC.ts
   • Ape/Brownie → test_patterns/ape.py, PoC filename: POC.py

5. Verify the test directory — check build config for where tests live

6. Create finding list with line offsets for the report file:

   ID	Title	Severity	Validity	Affected File(s)	Report Lines
   ...	...	...	...	...	{start}-{end}

7. Derive report slug — compact directory name from report filename and commit hash: {description}-{commit_hash} (e.g., savant-ai-scan-1eb2f41d)

8. Create output directory {test_dir}/audit_review/{report_slug}/findings/

9. Initialize STATE.md — write to {test_dir}/audit_review/{report_slug}/STATE.md using the State Protocol format

10. Return a single line: SETUP_COMPLETE|{state_md_path}|{framework}|{test_pattern_path}|{poc_filename}|{total_findings}|{report_slug}

After Setup Agent Returns

The orchestrator:

1. Parses the 1-line result to extract state_md_path, framework, test_pattern_path, poc_filename, total_findings, report_slug
2. Requests write permissions for {test_dir}/audit_review/{report_slug}/
3. Proceeds directly to Phase 1 — do not re-read the audit report or explore the codebase

Phase 1: Dispatch and Track

State Protocol

All session state lives on disk in {test_dir}/audit_review/{report_slug}/STATE.md. The orchestrator reads this file at the start of each batch cycle. This keeps orchestrator context flat regardless of how many findings are processed.

STATE.md format:

# Session State

**Report:** {report_path}
**Commit:** {hash}
**Test Pattern:** {test_pattern_path}
**Framework:** {Foundry | Hardhat | Ape}
**Total Findings:** {N}
**Processed:** {count}
**Remaining:** {count}

## Processed Findings

{ID}|{Status}|{Auditor Severity}|{Assessed Severity}|{Score}|{Validity Conf}|{Severity Conf}|{Summary}[|{validation notes if any}]
...

## Remaining Findings

| ID  | Title | Severity | Validity | Report Lines |
| --- | ----- | -------- | -------- | ------------ |
| ... | ...   | ...      | ...      | ...          |

Dispatch Loop — Background Agents with Disk-Based State

All 3 agents run in background. The orchestrator detects completion by reading RESULT files from disk — never by calling TaskOutput (which dumps agent transcripts into orchestrator context, causing context explosion).

Dispatch Cycle

1. Read STATE.md — get the current processed/remaining state
2. Count existing RESULT files — run ls findings/*/RESULT 2>/dev/null | wc -l to establish baseline count before this batch
3. Mark IN_PROGRESS — for up to 3 findings from Remaining (respecting grouping rules), move them from the Remaining table to the Processed section with status IN_PROGRESS:

   {ID}|IN_PROGRESS|||||||dispatched
   

4. Dispatch all 3 agents in background (run_in_background: true) — construct each prompt from SUBAGENT_PROMPT.md with template variables:
   • {test_pattern_path} → absolute path to the framework's test pattern file
   • {severity_reference_path} → absolute path to SEVERITY_REFERENCE.md
   • {output_dir} → {test_dir}/audit_review/{report_slug}/findings/{finding_id}/
   • {report_path} → path to audit report
   • {report_lines} → line range for this finding
   • {finding_id} → the finding's ID
   • {report_name} → name of the audit report
   • {poc_filename} → PoC test filename for the detected framework (POC.sol, POC.ts, or POC.py)
   • {processed_findings} → pipe-delimited lines from STATE.md (for duplicate detection)
5. Launch a background file watcher (run_in_background: true) — a single Bash command that exits when 2+ new RESULT files appear:

   target=$((existing + 2))
   while true; do
     current=$(ls findings/*/RESULT 2>/dev/null | wc -l)
     if [ "$current" -ge "$target" ]; then
       echo "READY: $current results (target was $target)"
       break
     fi
     sleep 30
   done
   

   Replace existing with the actual baseline count. This needs one approval per batch, then runs silently.
6. Do bookkeeping while waiting — prepare next batch's prompts from STATE.md, pre-create output directories with mkdir -p
7. Wait for watcher — call TaskOutput(watcher_task_id, block=true, timeout=600000). This blocks until 2+ agents complete. No approval needed.
8. Read new RESULT files — Glob findings/*/RESULT, read each new one, match against IN_PROGRESS entries
9. Update STATE.md — replace IN_PROGRESS lines with results, update Processed/Remaining counters
10. Refill and repeat — dispatch next batch to refill slots to 3

Why This Architecture

Background File Watcher

Instead of foreground polling (which requires per-poll approval and inflates context), use a single background bash command per batch cycle. Launch it with run_in_background: true immediately after dispatching agents. Check with TaskOutput (no approval needed) after doing bookkeeping.

Never use foreground sleep commands for polling. Each foreground sleep requires manual approval and adds messages to context. The background watcher is invisible to both.

Grouping Rules

• Findings touching the same file go in different dispatch slots (avoid conflicting writes)
• Findings touching different files can share concurrent slots
• Never exceed 3 concurrent agents

Post-Batch Validation

After collecting RESULT files for each batch, verify artifact completeness on disk:

Validation order: Check RESULT first (agent completed), then markdown artifact (most important deliverable), then PoC file. Subagents are instructed to write markdown before PoC, so a missing PoC with a present ISSUE.md/DISPUTE.md is more recoverable than the reverse.

For any missing artifact, append MISSING:{filename} to the finding's result line in STATE.md. If more than 2 findings in a batch have missing artifacts, investigate whether the subagent prompt template variables were filled correctly before continuing.

Context Management

• The orchestrator's context stays flat: it only holds STATE.md contents + the current batch's 1-line results
• Agent results are 1-line summaries — full artifacts are on disk
• If context pressure grows, stop and suggest continuing in a new session with the remaining STATE.md
• If an agent fails, log the failure in STATE.md and continue — do not retry

Phase 2: Final Scorecard

After all findings are processed, generate {test_dir}/audit_review/{report_slug}/SCORECARD.md.

Read all result lines from STATE.md. For confirmed findings, read the ISSUE.md to get quality dimension scores.

Self-identified invalid findings: If the audit report includes auditor validity labels (e.g., Validity: Invalid), findings the auditor self-identified as invalid are excluded from the quality score and listed in a separate section. These represent correct self-assessment and should not penalize audit quality. Add a Self-identified invalid (correct) row to the category table with the count, split the findings summary into scored and self-identified sections, and compute the score over only the scored findings.

Scorecard structure:

# Audit Review Scorecard: {Report Name}

**Commit:** {hash}
**Total Findings:** {N} ({scored} scored, {self_invalid} self-identified invalid)
**Audit Quality Score: {XX.XX} / 100** (scored findings only)

| Category                        | Count | Percentage |
| ------------------------------- | ----- | ---------- |
| Confirmed (severity accurate)   | X     | X%         |
| Confirmed (severity overstated) | X     | X%         |
| Disputed                        | X     | X%         |
| Duplicate                       | X     | X%         |
| Self-identified invalid (correct)| X     | —          |

## Scored Findings Summary

Sorted: Disputed first, then confirmed by assessed severity (Critical → Low).
Only includes findings the auditor did not self-identify as invalid.

| ID  | Title | Auditor Severity | Status | Assessed Severity | Finding Score | Validity Confidence | Severity Confidence |
| --- | ----- | ---------------- | ------ | ----------------- | ------------- | ------------------- | ------------------- |
| ... | ...   | ...              | ...    | ...               | ...           | ...                 | ...                 |

## Score Calculation

Severity weights: Critical=8, High=4, Medium=2, Low=1, Info=0.
Weight = max(auditor severity weight, assessed severity weight).
Formula: contribution = (weight / total_weight) × 100 × (score / 5)
Only scored findings are included — self-identified invalid findings are excluded.

| ID        | Severity Weight | Share of 100 | Finding Score | Contribution |
| --------- | --------------- | ------------ | ------------- | ------------ |
| ...       | ...             | ...          | ...           | ...          |
| **Total** | {total}         |              |               | **{final}**  |

Output Directory Structure

{test_dir}/audit_review/
  {report_slug}/                    ← one directory per report (e.g., savant-ai-scan-1eb2f41d/)
    STATE.md                        ← session state (orchestrator reads/writes)
    SCORECARD.md                    ← final scorecard (Phase 2)
    findings/
      {finding_id}/
        RESULT                      ← 1-line completion signal (agent writes, orchestrator reads)
        POC.{ext}                   ← confirmed AND disputed findings (NOT duplicates)
        ISSUE.md                    ← confirmed findings
        DISPUTE.md                  ← disputed findings
        DUPLICATE.md                ← duplicate findings (no POC needed)

Common Mistakes

Quality Checklist

Before completing the review:

• STATE.md reflects all findings processed
• All findings processed — none skipped
• Each finding directory has the correct artifact (ISSUE.md, DISPUTE.md, or DUPLICATE.md)
• Confirmed findings have POC.{ext} files
• SCORECARD.md generated with accurate counts
• Score calculations use correct severity weights
• Audit Quality Score computed correctly
• All agent summaries include self-validation results (no agents skipped it)
• No agents reported artifact naming fixes (if any did, investigate prompt clarity)
• Ban violations reviewed (acceptable or noted for skill improvement)

Converted and distributed by TomeVault — claim your Tome and manage your conversions.