Santa Method

Santa Method

Multi-agent adversarial verification framework. Make a list, check it twice. If it's naughty, fix it until it's nice.

The core insight: a single agent reviewing its own output shares the same biases, knowledge gaps, and systematic errors that produced the output. Two independent reviewers with no shared context break this failure mode.

When to Activate

Invoke this skill when:

• Output will be published, deployed, or consumed by end users
• Compliance, regulatory, or brand constraints must be enforced
• Code ships to production without human review
• Content accuracy matters (technical docs, educational material, customer-facing copy)
• Batch generation at scale where spot-checking misses systemic patterns
• Hallucination risk is elevated (claims, statistics, API references, legal language)

Do NOT use for internal drafts, exploratory research, or tasks with deterministic verification (use build/test/lint pipelines for those).

Architecture

┌─────────────┐
│  GENERATOR   │  Phase 1: Make a List
│  (Agent A)   │  Produce the deliverable
└──────┬───────┘
       │ output
       ▼
┌──────────────────────────────┐
│     DUAL INDEPENDENT REVIEW   │  Phase 2: Check It Twice
│                                │
│  ┌───────────┐ ┌───────────┐  │  Two agents, same rubric,
│  │ Reviewer B │ │ Reviewer C │  │  no shared context
│  └─────┬─────┘ └─────┬─────┘  │
│        │              │        │
└────────┼──────────────┼────────┘
         │              │
         ▼              ▼
┌──────────────────────────────┐
│        VERDICT GATE           │  Phase 3: Naughty or Nice
│                                │
│  B passes AND C passes → NICE  │  Both must pass.
│  Otherwise → NAUGHTY           │  No exceptions.
└──────┬──────────────┬─────────┘
       │              │
    NICE           NAUGHTY
       │              │
       ▼              ▼
   [ SHIP ]    ┌─────────────┐
               │  FIX CYCLE   │  Phase 4: Fix Until Nice
               │              │
               │ iteration++  │  Collect all flags.
               │ if i > MAX:  │  Fix all issues.
               │   escalate   │  Re-run both reviewers.
               │ else:        │  Loop until convergence.
               │   goto Ph.2  │
               └──────────────┘

Phase Details

Phase 1: Make a List (Generate)

Execute the primary task. No changes to your normal generation workflow. Santa Method is a post-generation verification layer, not a generation strategy.

# The generator runs as normal
output = generate(task_spec)

Phase 2: Check It Twice (Independent Dual Review)

Spawn two review agents in parallel. Critical invariants:

1. Context isolation — neither reviewer sees the other's assessment
2. Identical rubric — both receive the same evaluation criteria
3. Same inputs — both receive the original spec AND the generated output
4. Structured output — each returns a typed verdict, not prose

REVIEWER_PROMPT = """
You are an independent quality reviewer. You have NOT seen any other review of this output.

## Task Specification
{task_spec}

## Output Under Review
{output}

## Evaluation Rubric
{rubric}

## Instructions
Evaluate the output against EACH rubric criterion. For each:
- PASS: criterion fully met, no issues
- FAIL: specific issue found (cite the exact problem)

Return your assessment as structured JSON:
{
  "verdict": "PASS" | "FAIL",
  "checks": [
    {"criterion": "...", "result": "PASS|FAIL", "detail": "..."}
  ],
  "critical_issues": ["..."],   // blockers that must be fixed
  "suggestions": ["..."]         // non-blocking improvements
}

Be rigorous. Your job is to find problems, not to approve.
"""

Dispatch both reviewers in the same turn so they run concurrently. In Claude Code, that means two Agent tool calls in a single message (subagent_type: "general-purpose", descriptions "Santa Reviewer B" and "Santa Reviewer C"), each receiving only its own filled-in REVIEWER_PROMPT. Each subagent starts from an empty context — neither knows the other exists.

Rubric Design

The rubric is the most important input. Vague rubrics produce vague reviews. Every criterion must have an objective pass/fail condition.

Domain-Specific Rubric Extensions

Content/Marketing:

• Brand voice adherence
• SEO requirements met (keyword density, meta tags, structure)
• No competitor trademark misuse
• CTA present and correctly linked

Code:

• Type safety (no any leaks, proper null handling)
• Error handling coverage
• Security (no secrets in code, input validation, injection prevention)
• Test coverage for new paths

Compliance-Sensitive (regulated, legal, financial):

• No outcome guarantees or unsubstantiated claims
• Required disclaimers present
• Approved terminology only
• Jurisdiction-appropriate language

Phase 3: Naughty or Nice (Verdict Gate)

def santa_verdict(review_b, review_c):
    """Both reviewers must pass. No partial credit."""
    if review_b.verdict == "PASS" and review_c.verdict == "PASS":
        return "NICE"  # Ship it

    # Merge flags from both reviewers, deduplicate
    all_issues = dedupe(review_b.critical_issues + review_c.critical_issues)
    all_suggestions = dedupe(review_b.suggestions + review_c.suggestions)

    return "NAUGHTY", all_issues, all_suggestions

Why both must pass: if only one reviewer catches an issue, that issue is real. The other reviewer's blind spot is exactly the failure mode Santa Method exists to eliminate.

Phase 4: Fix Until Nice (Convergence Loop)

MAX_ITERATIONS = 3

for iteration in range(MAX_ITERATIONS):
    verdict, issues, suggestions = santa_verdict(review_b, review_c)

    if verdict == "NICE":
        log_santa_result(output, iteration, "passed")
        return ship(output)

    # Fix all critical issues (suggestions are optional)
    output = fix_agent.execute(
        output=output,
        issues=issues,
        instruction="Fix ONLY the flagged issues. Do not refactor or add unrequested changes."
    )

    # Re-run BOTH reviewers on fixed output (fresh agents, no memory of previous round)
    review_b = spawn_fresh_reviewer("Santa Reviewer B", REVIEWER_PROMPT, output)
    review_c = spawn_fresh_reviewer("Santa Reviewer C", REVIEWER_PROMPT, output)

# Exhausted iterations — escalate
log_santa_result(output, MAX_ITERATIONS, "escalated")
escalate_to_human(output, issues)

Critical: each review round uses fresh agents. Reviewers must not carry memory from previous rounds, as prior context creates anchoring bias.

Implementation Patterns

Pattern A: Claude Code Subagents (Recommended)

Subagents provide true context isolation. Each reviewer is a separate process with no shared state.

How to run it with the Agent tool:

1. Issue two Agent tool calls in a single message so they run in parallel — subagent_type: "general-purpose", with description set to "Santa Reviewer B" and "Santa Reviewer C".
2. Each call's prompt contains the full filled-in reviewer prompt (task spec + rubric + output under review) and nothing else. A subagent starts with empty context, so neither reviewer knows the other exists.
3. Collect both structured verdicts and apply the verdict gate: both must pass before the output ships.
4. On every fix round, issue brand-new Agent calls — never message or reuse a previous reviewer. Fresh agents each round eliminate anchoring from earlier verdicts.

Pattern B: Sequential Inline (Fallback)

When subagents aren't available, simulate isolation with explicit context resets:

1. Generate output
2. New context: "You are Reviewer 1. Evaluate ONLY against this rubric. Find problems."
3. Record findings verbatim
4. Clear context completely
5. New context: "You are Reviewer 2. Evaluate ONLY against this rubric. Find problems."
6. Compare both reviews, fix, repeat

The subagent pattern is strictly superior — inline simulation risks context bleed between reviewers.

Pattern C: Batch Sampling

For large batches (100+ items), full Santa on every item is cost-prohibitive. Use stratified sampling:

1. Run Santa on a random sample (10-15% of batch, minimum 5 items)
2. Categorize failures by type (hallucination, compliance, completeness, etc.)
3. If systematic patterns emerge, apply targeted fixes to the entire batch
4. Re-sample and re-verify the fixed batch
5. Continue until a clean sample passes

import random

def santa_batch(items, rubric, sample_rate=0.15):
    sample = random.sample(items, max(5, int(len(items) * sample_rate)))

    for item in sample:
        result = santa_full(item, rubric)
        if result.verdict == "NAUGHTY":
            pattern = classify_failure(result.issues)
            items = batch_fix(items, pattern)  # Fix all items matching pattern
            return santa_batch(items, rubric)   # Re-sample

    return items  # Clean sample → ship batch

Failure Modes and Mitigations

Integration with Other Skills

Metrics

Track these to measure Santa Method effectiveness:

• First-pass rate: % of outputs that pass Santa on round 1 (target: >70%)
• Mean iterations to convergence: average rounds to NICE (target: <1.5)
• Issue taxonomy: distribution of failure types (hallucination vs. completeness vs. compliance)
• Reviewer agreement: % of issues flagged by both reviewers vs. only one (low agreement = rubric needs tightening)
• Escape rate: issues found post-ship that Santa should have caught (target: 0)

Cost Analysis

Santa Method costs approximately 2-3x the token cost of generation alone per verification cycle. For most high-stakes output, this is a bargain:

Cost of Santa = (generation tokens) + 2×(review tokens per round) × (avg rounds)
Cost of NOT Santa = (reputation damage) + (correction effort) + (trust erosion)

For batch operations, the sampling pattern reduces cost to ~15-20% of full verification while catching >90% of systematic issues.