Ralph Review

Ralph Review

Post-build verification and remediation skill. Run this after Ralph finishes a batch of iterations (or an entire service build) to catch issues before they compound.

Overview

Three-stage process:

1. Verify — parallel agents audit what Ralph produced
2. Cross-examine — agents see each other's findings and react
3. Remediate — fix HIGH findings, verify fixes with trailing agents

Input Detection

Parse the user's input for:

• Service name or phase: which service/phase Ralph was working on (e.g., rideshare, tesla, dashboard, or phase-1)
• Iteration range: e.g., "10-20" means audit iterations 10 through 20 (optional — if omitted, audit everything since last review)

Resolve Ralph artifact paths by searching:

1. First try: mock-apis/{service}/ralph/prd/prd-mock-{service}.json
2. Then try: {service}/ralph/prd/*.json
3. Then try: Plan/phase-{N}-*/ralph/prd/prd-phase-{N}.json
4. Fallback: glob for **/ralph/prd/prd*.json and let user pick

Stage 0: Build Project Context

Before launching any agents, you MUST build a context block from the actual codebase.

Read these files (in order, skip if missing):

1. AGENTS.md — canonical patterns, conventions, tech stack
2. CLAUDE.md — project instructions, dev commands
3. package.json (root) — workspace structure, scripts, dependencies
4. The PRD file for this service/phase — stories, acceptance criteria

From these, construct a {context block} that includes:

• Tech stack (framework, language, package manager, monorepo tool if any)
• Project structure (apps, packages, key directories)
• Auth/security patterns (if documented in AGENTS.md)
• Key conventions (import patterns, naming, file organization)
• Dev commands (typecheck, lint, build, test)
• Quality gate commands (what to run and what "pass" looks like)
• Commit message format (from AGENTS.md if documented)

This context block gets injected into every agent prompt below where {context block} appears.

Stage 1: Quick Gates (Sequential)

Run these first. If they fail, fix before proceeding to deeper audit.

Detect the correct commands from package.json scripts or CLAUDE.md:

Gate 1: Typecheck

# Use whichever exists: pnpm typecheck, npm run typecheck, npx tsc --noEmit
{TYPECHECK_COMMAND} 2>&1 | tail -30

If FAIL -> fix immediately before proceeding.

Gate 2: Tests

# Use whichever exists: pnpm test, npm test, npx vitest run
{TEST_COMMAND} 2>&1 | tail -50

If FAIL -> fix immediately before proceeding. Tests are a hard gate — they must pass.

Gate 3: Lint (if available)

# Use whichever exists: pnpm lint, npm run lint, npx eslint .
{LINT_COMMAND} 2>&1 | tail -30

If FAIL -> fix immediately before proceeding. If no lint script exists, skip.

Gate 4: PRD Status

node -e "
const prd = require('{PRD_FILE_PATH}');
const stories = prd.stories;
const complete = stories.filter(s => s.status === 'complete');
const incomplete = stories.filter(s => !['complete','deferred'].includes(s.status));
console.log('Complete: ' + complete.length + '/' + stories.length);
console.log('Remaining: ' + incomplete.length);
incomplete.slice(0, 10).forEach(s => console.log('  - ' + s.id + ': ' + s.title));
"

Report gate results to user before proceeding:

Quick Gates:
  Typecheck: {PASS/FAIL}
  Tests:     {PASS/FAIL} ({X passed, Y failed})
  Lint:      {PASS/FAIL/SKIPPED}
  PRD:       {X/Y} complete, {Z} remaining

Stage 2: Parallel Audit Agents (6 agents, all background)

Launch ALL 6 agents simultaneously using run_in_background: true.

• All agents: model: "sonnet" — use Sonnet for all agents, never Haiku
• All agents use subagent_type: "Explore"

CRITICAL: Every agent prompt MUST include the context block built in Stage 0.

CRITICAL: Every agent prompt MUST include the Agent Roster — a list of all 6 agents and their focus areas. This lets each agent flag cross-cutting concerns for other agents to verify in Stage 2.5.

Include this in every agent prompt:

## Agent Roster (you are Agent {N})
1. Security Auditor — auth, access control, injection, secrets, HMAC
2. Architecture Auditor — patterns, conventions, file organization, dependencies
3. Data Layer Auditor — schema, queries, indexes, migrations, tenant isolation
4. UI/UX Auditor — pages, components, accessibility, loading/error/empty states
5. Integration Auditor — cross-package wiring, type flow, env vars, build pipeline
6. Gotcha Hunter — TODOs, hardcoded values, race conditions, missing awaits, console.log

If you find something outside your domain, note it as:
  CROSS-REF [Agent N]: [what they should check]

Agent 1: Security Auditor (Model: sonnet)

{context block}

{agent roster}

## Your Focus: Security Audit

Audit all code produced by Ralph in this service/phase for security issues:

1. If an RBAC matrix or role-based access control exists, verify ALL new procedures are registered. Default-deny means unregistered = broken.
2. Check auth middleware — verify tenant isolation / org scoping is still wired correctly.
3. For every new backend route or procedure, verify:
   - All mutations require authentication
   - Input validation uses schema validation (Zod, Joi, etc.), not raw input
   - No raw SQL without parameterization (string concatenation in queries)
4. For every new API route or webhook handler, verify:
   - Webhook routes validate signatures (HMAC, etc.)
   - Auth checks present
   - No secrets in code
5. Check for:
   - process.env usage in frontend/UI packages (should use typed env config)
   - Error messages that leak internals in production (must guard with NODE_ENV)
   - Missing CSRF/CORS protections on mutations
   - timingSafeEqual for any signature comparisons (not === )
   - Secrets or tokens in URL parameters

## OUTPUT FORMAT (MANDATORY)
SECURITY AUDIT REPORT
=====================
Scope: [service/iterations audited]
Findings: [count by severity]

HIGH:
- [H-NNN]: [title] | [file:line] | [description]

MEDIUM:
- [M-NNN]: [title] | [file:line] | [description]

LOW:
- [L-NNN]: [title] | [file:line] | [description]

CROSS-REF:
- [Agent N]: [what they should verify]

Auth Coverage: [summary of auth/RBAC status]
Webhook Security: [all validated / gaps / N/A]

Agent 2: Architecture Auditor

{context block}

{agent roster}

## Your Focus: Architecture & Pattern Conformance

Verify Ralph followed established patterns:

1. Read AGENTS.md (or CLAUDE.md) — these define the canonical patterns
2. For every new file Ralph created, verify:
   - Files are in the correct directories per project conventions
   - Import conventions are followed (package aliases, no cross-boundary relative imports)
   - Established patterns are used (not reinvented)
3. Check dependency graph:
   - No circular dependencies between packages/modules
   - Dependency direction matches architecture (leaf packages don't import from app layer)
4. Naming conventions followed (routes, files, variables)
5. New modules/routers/services registered in their respective index/root files

## OUTPUT FORMAT (MANDATORY)
ARCHITECTURE AUDIT REPORT
=========================
Scope: [service/iterations audited]
Findings: [count by severity]

HIGH:
- [H-NNN]: [title] | [file] | [description]

MEDIUM:
- [M-NNN]: [title] | [file] | [description]

LOW:
- [L-NNN]: [title] | [file] | [description]

CROSS-REF:
- [Agent N]: [what they should verify]

Pattern Conformance: [PASS/PARTIAL/FAIL]
Dependency Graph: [CLEAN/VIOLATIONS]
Registration: [all registered / total new modules]

Agent 3: Data Layer Auditor

{context block}

{agent roster}

## Your Focus: Database & Data Integrity

Audit schema changes, migrations, and data access patterns:

1. Read the schema file(s) — check for:
   - Proper column types (numeric precision, enum usage)
   - FK relationships defined
   - Indexes on frequently queried columns
   - Tenant isolation columns on all tenant-scoped tables (if multi-tenant)
2. Read any new migration files:
   - Wrapped in transactions (BEGIN/COMMIT)?
   - Idempotent (IF NOT EXISTS, IF EXISTS)?
   - Data migration for column type changes?
3. Check query patterns in backend routes/services:
   - Tenant-scoped queries include tenant filter
   - Pagination present on list queries (cursor-based preferred over offset)
   - No N+1 patterns (batch operations, not loops)
   - Proper error handling on DB operations
4. Type helpers / inference types exported where applicable

## OUTPUT FORMAT (MANDATORY)
DATA LAYER AUDIT REPORT
=======================
Scope: [service/iterations audited]
Findings: [count by severity]

HIGH:
- [H-NNN]: [title] | [file:line] | [description]

MEDIUM:
- [M-NNN]: [title] | [file:line] | [description]

LOW:
- [L-NNN]: [title] | [file:line] | [description]

CROSS-REF:
- [Agent N]: [what they should verify]

Schema Integrity: [PASS/PARTIAL/FAIL]
Migration Quality: [PASS/PARTIAL/FAIL]
Tenant Isolation: [PASS/PARTIAL/FAIL/N/A]

Agent 4: UI/UX Auditor

{context block}

{agent roster}

## Your Focus: UI/UX Quality

**Skip this audit entirely if the project has no frontend/UI layer. Report "N/A — no frontend" and exit.**

Audit frontend code for consistency and completeness:

1. For every new page:
   - Does it have a loading state (skeleton, spinner, loading.tsx)?
   - Does it have proper empty state handling?
   - Does it have error boundary coverage?
2. For every new component:
   - Uses the project's UI component library (not raw HTML for things the library covers)
   - Dark mode compatible (CSS variables, not hardcoded colors) — if applicable
   - Accessible (aria-labels on interactive elements)
   - Mobile-responsive (no fixed widths)
3. Check navigation — are new pages linked in nav/sidebar?
4. Forms use the project's form pattern (e.g., react-hook-form + Zod, Formik, etc.)
5. User feedback via toast/notification utility (not window.alert)
6. Error messages use dev-only guards for stack traces

## OUTPUT FORMAT (MANDATORY)
UI/UX AUDIT REPORT
==================
Scope: [service/iterations audited]
Findings: [count by severity]

HIGH:
- [H-NNN]: [title] | [file] | [description]

MEDIUM:
- [M-NNN]: [title] | [file] | [description]

LOW:
- [L-NNN]: [title] | [file] | [description]

CROSS-REF:
- [Agent N]: [what they should verify]

Pages Audited: [count]
Missing loading states: [list or "none"]
Missing empty states: [list or "none"]
Accessibility: [PASS/PARTIAL/FAIL]

Agent 5: Integration Auditor

{context block}

{agent roster}

## Your Focus: Cross-Package Integration

Verify everything wires together correctly:

1. For every new backend route/procedure:
   - Registered in the root router/index
   - Registered in access control (RBAC matrix, permissions, etc.) if applicable
   - Input schemas importable from shared types package
2. For every new page that calls the backend:
   - Correct API paths / procedure names
   - Error handling for API errors
   - Loading states while queries resolve
3. Cross-package type safety:
   - DB types flow to backend output types
   - Backend output types flow to frontend components
   - No `any` types at package boundaries
4. Environment variables:
   - New vars added to both typed env config AND .env.example
   - Optional vs required matches actual usage
5. API contract compliance (if api-contract.md or similar exists):
   - Response shapes match the contract spec
   - Status codes match documented behavior
   - Error formats are consistent
6. Build pipeline:
   - New packages detected by build tool (turbo, nx, etc.)
   - No imports that would break at build time

## OUTPUT FORMAT (MANDATORY)
INTEGRATION AUDIT REPORT
========================
Scope: [service/iterations audited]
Findings: [count by severity]

HIGH:
- [H-NNN]: [title] | [file] | [description]

MEDIUM:
- [M-NNN]: [title] | [file] | [description]

LOW:
- [L-NNN]: [title] | [file] | [description]

CROSS-REF:
- [Agent N]: [what they should verify]

Type Flow: [PASS/PARTIAL/FAIL]
Contract Compliance: [PASS/PARTIAL/FAIL/N/A]
Registration: [complete/incomplete]
Env Var Sync: [synced/out-of-sync]

Agent 6: Gotcha Hunter

{context block}

{agent roster}

## Your Focus: Edge Cases & Undocumented Issues

Find things the other auditors might miss:

1. Search for TODO/FIXME/HACK/XXX comments in all modified files
2. Look for hardcoded values (URLs, ports, limits, magic numbers)
3. Check for race conditions in async code
4. Look for missing error handling (unhandled promise rejections)
5. Check git log for patterns of Ralph fixing its own mistakes
6. Look for commented-out code that should be removed
7. Check for console.log statements that should be console.error or removed
8. Verify no secrets or sensitive data in code
9. Check for N+1 query patterns in loops
10. Look for missing `await` on async calls

## OUTPUT FORMAT (MANDATORY)
GOTCHA REPORT
=============
Scope: [service/iterations audited]
Issues Found: [count]

Gotchas:
- [severity]: [file:line] [description]

Hardcoded Values:
- [file:line]: [value] [risk]

TODO/FIXME Found:
- [file:line]: [text]

Console.log Cleanup:
- [file:line]: [should be error/removed]

Race Conditions:
- [file:line]: [description] or "none found"

CROSS-REF:
- [Agent N]: [what they should verify]

Stage 2.5: Cross-Examination Round

After all 6 agents complete, orchestrate a cross-examination.

1. Collect all 6 reports
2. Extract all CROSS-REF items — these are questions agents have for each other
3. Also identify overlapping findings — where 2+ agents flagged the same area
4. Also identify blind spots — areas only one agent covered that deserve a second opinion

For each substantive cross-reference or overlap, launch a targeted verification agent:

subagent_type: "Explore"
model: "sonnet"
run_in_background: true

Prompt pattern:

{context block}

## Cross-Examination Task

Agent {N} ({domain}) found:
  {finding summary with file paths}

Agent {M} ({domain}) is asked to verify:
  {the cross-ref question}

Additional context from other agents:
  {any related findings from other agents on the same files/areas}

Your job: Read the relevant files and either CONFIRM, REFUTE, or EXPAND the finding.

## OUTPUT FORMAT (MANDATORY)
CROSS-EXAM RESULT
=================
Original Finding: [H/M/L-NNN from Agent N]
Verdict: [CONFIRMED / REFUTED / EXPANDED]
Evidence: [what you found]
Revised Severity: [same / upgraded / downgraded]
Additional Finding: [if EXPANDED, the new issue] or "none"

Guidelines for cross-examination:

• Don't cross-examine every finding — only substantive ones (HIGH findings, CROSS-REFs, and overlaps)
• Batch related cross-refs into single agents when they touch the same files
• Limit to ~4 cross-exam agents max to keep costs reasonable
• If no CROSS-REFs and no overlaps, skip this stage entirely

Stage 3: Consolidate & Report

After all agents (including cross-exam) complete, consolidate findings:

1. Collect all agent reports + cross-exam verdicts
2. Adjust severities based on cross-exam (confirmed = higher confidence, refuted = drop)
3. Deduplicate (same issue found by multiple agents = higher confidence)
4. Number findings globally: H-001, H-002, M-001, M-002, L-001, etc.
5. Present consolidated report to user:

Ralph Review — {Service/Phase} Complete
========================================

Quick Gates: Typecheck {PASS/FAIL} | Tests {PASS/FAIL} | Lint {PASS/FAIL} | PRD {X/Y}

Findings Summary:
  HIGH:   {count} (must fix)
  MEDIUM: {count} (should fix)
  LOW:    {count} (nice to have)

Cross-Agent Confirmations (highest confidence):
  {findings confirmed by cross-examination or flagged by 2+ agents}

HIGH Findings:
  H-001: [title] — [file] — [one-line description]
  H-002: [title] — [file] — [one-line description]

MEDIUM Findings:
  M-001: [title] — [file] — [one-line description]

LOW Findings:
  L-001: [title] — [file] — [one-line description]

Potential Phantom Findings (refuted or suspicious):
  {findings that cross-exam refuted or couldn't verify}

Recommendation: [Fix HIGH items now / All clear / etc.]

Stage 4: Remediation (User Approval Required)

Ask the user before fixing anything. Present the findings and ask:

• "Fix all HIGH items?" (recommended)
• "Fix HIGH + MEDIUM?"
• "Skip fixes, just report?"

If approved:

1. Spin up fix agents — one per HIGH finding (or batch related findings)
   • Each fix agent is subagent_type: "general-purpose", run_in_background: true
   • Prompt includes the specific finding, file path, and what to fix
   • Agent reads the file, makes the fix, does NOT run tests
2. Wait for all fix agents to complete
3. Run gates again — typecheck + tests + lint
4. Trailing verification agents — spin up 1-2 Explore agents that re-audit the specific files/areas that were changed by fix agents. These are NOT full re-audits — they're scoped to the fix areas only:

subagent_type: "Explore"
model: "sonnet"
run_in_background: true

Prompt pattern:

{context block}

## Post-Fix Verification

The following fixes were just applied:
{list of findings fixed, with file paths and what changed}

Your job:
1. Read each modified file
2. Verify the fix actually resolves the issue (not just a surface patch)
3. Check that the fix didn't introduce NEW issues (regressions, type errors, broken imports)
4. Check that the fix follows project conventions (from AGENTS.md)

## OUTPUT FORMAT (MANDATORY)
FIX VERIFICATION REPORT
=======================
Verified: [count]
Regressions: [count or "none"]

Per-Fix:
- [H/M-NNN]: [GOOD / REGRESSION / INCOMPLETE]
  Evidence: [what you found]

New Issues Introduced:
- [description] or "none"

5. Report results to user (including verification agent findings)
6. If regressions found — fix them, re-run gates, re-verify (limit to 2 cycles)
7. Commit if all clear (with user approval)

Commit message format (adapt to project conventions from AGENTS.md):

[AUDIT]: {service/scope} — {count} findings resolved

{one-line per finding fixed}

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Stage 5: Update Progress

Update the progress file with audit results:

## [DATE] - Ralph Review ({Service}, Iterations X-Y)
- Quick Gates: Typecheck PASS, Tests PASS, Lint PASS, PRD X/Y
- Findings: H HIGH, M MEDIUM, L LOW
- Cross-Exam: {N} findings verified, {M} refuted
- Fixed: {list of fixed findings}
- Deferred: {list of deferred findings with reasons}

Key Principles

1. Parallel agents, cross-examination, sequential fixes — audit fast, verify findings, fix carefully
2. Agents know about each other — the Agent Roster and CROSS-REF system lets agents flag cross-cutting concerns for peers to verify
3. Trailing verifiers catch regressions — fix agents can introduce new bugs; verification agents catch them
4. Phantom finding awareness — agents may cite wrong line numbers or non-existent issues. Cross-examination helps filter these out.
5. Never skip gates — typecheck, tests, and lint MUST pass before deeper audit AND after fixes
6. User approves all fixes — present findings, ask before modifying code
7. Commit granularly — one commit for audit fixes, separate from Ralph's work