Code Audit Methodology

Code Audit Methodology

View Landing Page | GitHub

This document explains the research-backed principles behind Heavy3 Code Audit's design.

Table of Contents

1. The Synthesis Table - Our trademark feature
2. The Council - Three specialized reviewers
3. Why Multi-Model? - Research backing
4. Plan Review - Review before you code
5. Context Engineering - What context to send
6. Context Positioning - Lost in the Middle
7. Large Change Handling - 50+ files, 10K+ lines
8. Implementation Details - Code references
9. Academic References - Peer-reviewed sources

The Synthesis Table (Trademark Feature)

After council review, Claude synthesizes all findings into a 3-column comparison table—our trademark feature that differentiates Heavy3 Code Audit from other tools.

Example output:

Legend: ✅ = No issues | ⚠️ = Warning | ❌ = Critical issue

What you get:

• Consensus Issues - Problems flagged by 2+ reviewers (high confidence)
• Notable Findings - Unique insights from each specialist
• Final Recommendation - APPROVE / APPROVE WITH CHANGES / REQUEST CHANGES
• Priority Actions - Ranked list of fixes

Why this matters:

• Shows exactly where reviewers agree and disagree
• Provides transparency no other tool offers
• Helps you make informed decisions about which issues to address

The Council

Three specialized reviewers, each with web search:

Why Grok 4 for Security?

Grok 4 was selected as Security Analyst based on independent security benchmarks:

The Kilo AI test specifically evaluated Grok 4 on prototype pollution, agentic AI supply-chain attacks, and OS command injection—achieving fix quality scores of 83-85/100 with references to OWASP AI Top 10 and NIST AI RMF standards.

Three Pillars of Diversity:

You code with Claude. Our council (GPT + Gemini + Grok) catches what Claude misses.

Why Multi-Model?

Single models classify code correctness only ~68% of the time. Research on LLM code review accuracy shows significant room for improvement:

Every model has different blind spots. Sonar's Dec 2025 analysis of millions of lines of generated code:

The insight: The probability of two different architectures hallucinating the same bug is significantly lower than one. Using 3 specialized models catches what any single model misses.

What Context to Send for High-Quality Reviews

Research Finding: Quality of context matters more than quantity. Well-selected 8K-32K tokens outperforms noisy 100K+ contexts.

Based on LAURA (Zhang et al., IEEE ASE 2025) and CodeRabbit's context engineering:

Key Research Insight: "Incorporating problem descriptions into prompts consistently improved performance, highlighting the importance of code comments and pull request descriptions." (Evaluating LLMs for Code Review, 2025)

Context Positioning: The "Lost in the Middle" Problem

The Problem: LLMs exhibit a U-shaped attention curve where they best process information at the beginning and end of long contexts, while information in the middle is often neglected.

Research Source: Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (TACL 2024)

"Performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts."

Psychological Parallel: This mirrors the "serial-position effect" (Ebbinghaus, 1913) in human psychology.

Our Implementation (build_user_message() in review.py and council.py):

┌─────────────────────────────────────────┐
│ START: Critical context                  │
│   - Developer intent (conversation)      │  ← conversation_context first
│   - PR description (intent)              │  ← pr_metadata second
│   - Code diff (actual changes)           │  ← diff third
├─────────────────────────────────────────┤
│ MIDDLE: Supporting context               │
│   - Full file contents                   │  ← file_contents
│   - Documentation                        │  ← documentation
│   - Test files                           │  ← test_files
│   - Cross-file dependencies              │  ← dependent_files
├─────────────────────────────────────────┤
│ END: Instructions (in system prompt)     │
│   - Review focus areas                   │  ← SYSTEM_PROMPT
│   - Output format requirements           │
└─────────────────────────────────────────┘

Multi-Model Consensus: Why Three Reviewers

The Problem with Single-Model Review (Sonar LLM Leaderboard, Dec 2025):

Even top-tier models have significant blind spots. Per million lines of generated code:

Key Insight: Each model excels in different areas and fails in different ways. Multi-model consensus catches what any single model misses.

Research Support:

1. Hashgraph-Inspired Consensus (2025): Treating each model as a "peer" in a distributed consensus system achieves "blockchain-grade properties of consistency and Byzantine fault tolerance."

2. Disagreement as Data (Jan 2025): "LLM agents' semantic reasoning similarity robustly differentiates consensus from disagreement and correlates with human coding reliability."

Plan/Architecture Review: The Missing Piece

No major code review tool effectively reviews plans before implementation.

Most tools focus on implementation-level code review:

• Syntax errors, linting violations
• Bug detection after code is written
• Code style compliance

Heavy3 Code Audit uniquely offers pre-implementation validation:

Why this matters for AI-assisted coding (vibe coding):

• The plan determines 80% of success
• Catching design issues early saves massive rework
• Architecture review is where multi-model consensus shines

Usage:

/h3 plan.md           # Review a specific plan file
/h3                   # Smart detection finds plan.md in cwd or ~/.claude/plans/

Our Implementation (Verified)

JSON Structure Sent to Models:

{
  "review_type": "code|plan|pr",
  "conversation_context": {
    "original_request": "Add logout button to navbar",
    "approach_notes": "Using existing Button component",
    "relevant_exchanges": [
      {"role": "user", "content": "Can you add a logout button?"},
      {"role": "assistant", "content": "I'll add it using the existing Button component..."}
    ],
    "previous_review_findings": "Prior review suggested adding confirmation dialog"
  },
  "diff": "--- a/src/foo.ts\n+++ b/src/foo.ts\n...",
  "file_contents": {
    "src/foo.ts": "// Full file content..."
  },
  "test_files": {
    "src/foo.test.ts": "// Test content..."
  },
  "dependent_files": {
    "src/bar.ts": "import { foo } from './foo';\n...\nfoo(data);"
  },
  "documentation": {
    "CLAUDE.md": "// Project guidelines..."
  },
  "pr_metadata": {
    "number": 123,
    "title": "Add feature X",
    "body": "This PR adds..."
  }
}

Context Engineering Best Practices

Based on CodeRabbit's "Context Engineering" blog and our implementation:

1. Deduplicate Redundant Context: We use a single-pass context builder that avoids duplication.

2. Respect Token Budgets:

   • 200K tokens (max_context)
   • Graceful truncation with [... truncated due to length ...] marker

3. Include Developer Intent: PR descriptions and commit context included via pr_metadata.

4. Streaming for UX: Single mode streams tokens as they arrive.

5. Progress Indicators: Council shows completion status for each model.

Large Change Handling

For changes exceeding context limits (50+ files, 10K+ lines):

1. Detection: Count files and lines via git stats
2. Module Grouping: Break into logical modules
3. Sequential Review: Review each module with progress tracking
4. Cross-Module Summary: Final pass for dependencies

References

Peer-Reviewed Academic Work

Published in peer-reviewed venues (conferences, journals):

Academic Preprints

ArXiv preprints (not yet peer-reviewed):

Industry Analysis

Reports and analysis from industry sources:

Summary