Multi Model Validation

Multi-Model Validation

Version: 3.3.0 Purpose: Patterns for running multiple AI models in parallel via Claudish proxy with context-aware preferences, dynamic model discovery, session-based workspaces, and performance statistics Status: Production Ready

Overview

Multi-model validation is the practice of running multiple AI models (Grok, Gemini, GPT-5, DeepSeek, etc.) in parallel to validate code, designs, or implementations from different perspectives. This achieves:

• 3-5x speedup via parallel execution (15 minutes → 5 minutes)
• Consensus-based prioritization (issues flagged by all models are CRITICAL)
• Diverse perspectives (different models catch different issues)
• Cost transparency (know before you spend)
• Free model discovery (NEW v3.0) - find high-quality free models from trusted providers
• Performance tracking - identify slow/failing models for future exclusion
• Data-driven recommendations - optimize model shortlist based on historical performance

Key Innovations:

1. Context-Aware Preferences (NEW v3.3.0) - Automatically use saved model preferences per task type (debug/research/coding/review) from .claude/multimodel-team.json
2. Dynamic Model Discovery (v3.0) - Read the live catalog (list_models) for current available models (live, 24h cache)
3. Session-Based Workspaces (v3.0) - Each validation session gets a unique directory to prevent conflicts
4. 4-Message Pattern - Ensures true parallel execution by using only Agent tool calls in a single message
5. Pattern 7-8 - Statistics collection and data-driven model recommendations

This skill is extracted from the /review command and generalized for use in any multi-model workflow.

⚠️ MANDATORY: Learn and Reuse User Preferences

Model preferences are learned per context and reused automatically.

• First time a context is used → ASK user → SAVE to that context
• Next time same context → VALIDATE the saved IDs against list_models, then use the survivors automatically (no asking). "No asking" applies to the selection, never to the catalog check — saved IDs go stale and must be re-checked every run.
• User explicitly says "change models" or "different models" → ASK and UPDATE

# FIRST STEP - Read preferences file
cat .claude/multimodel-team.json 2>/dev/null

Flow:

1. Detect context from task keywords
   - "debug", "error", "bug", "fix" → debug
   - "research", "analyze", "investigate" → research
   - "implement", "build", "create", "code" → coding
   - "review", "audit", "check" → review

2. Check if contextPreferences[context] exists and is non-empty

   IF EXISTS (has models saved):
   → Call: list_models (claudish MCP) and KEEP ONLY the saved IDs it still lists
     Saved preferences are user policy, not a catalog snapshot — they go stale
     silently. This applies to defaultModels and contextPreferences alike; see
     claudish:claudish-usage → "Every field of the preferences file is untrusted"
   → Name every dropped ID in your reply
   → DO NOT ask the user to re-pick while at least one saved ID survives
   → If NOTHING survives, say so and offer live alternatives

   IF EMPTY/MISSING (first time for this context):
   → Call: list_models (claudish MCP — current models, pricing, capabilities)
   → Ask user to select models (AskUserQuestion)
   → Save to contextPreferences[context]
   → Proceed with validation

3. User override triggers (explicit request to change):
   - "use different models"
   - "change models"
   - "update model preferences"
   → Ask user to select new models
   → Update contextPreferences[context]

Example - Learning Flow:

# First debug task ever:
Task: "Debug this authentication error"
→ Context: debug
→ contextPreferences.debug is empty
→ ASK: "Which models for debug tasks?"
→ User selects: grok, glm, minimax
→ SAVE to contextPreferences.debug
→ Run with those models

# Second debug task:
Task: "Debug the API timeout"
→ Context: debug
→ contextPreferences.debug = ["grok", "glm", "minimax"]
→ USE directly (no asking)
→ Run with saved models

# User wants to change:
Task: "Debug this error, use different models"
→ Detected: "different models" override trigger
→ ASK: "Which models for debug tasks?"
→ User selects: gemini, LATEST_GPT_MODEL
→ UPDATE contextPreferences.debug
→ Run with new models

Related Skills

CRITICAL: Tracking Protocol Required

Before using any patterns in this skill, ensure you have completed the pre-launch setup from multimodel:model-tracking-protocol.

Launching models without tracking setup = INCOMPLETE validation.

Cross-References:

• multimodel:model-tracking-protocol - MANDATORY tracking templates and protocols (NEW in v0.6.0)
  • Pre-launch checklist (8 required items)
  • Tracking table templates
  • Failure documentation format
  • Results presentation template
• multimodel:quality-gates - Approval gates and severity classification
• multimodel:task-orchestration - Progress tracking during execution
• multimodel:error-recovery - Handling failures and retries

Skill Integration:

This skill (multi-model-validation) defines execution patterns (how to run models in parallel). The model-tracking-protocol skill defines tracking infrastructure (how to collect and present results).

Use both together:

skills: multimodel:multi-model-validation, multimodel:model-tracking-protocol

Core Patterns

Pattern 0: Session Setup and Model Discovery (NEW v3.0)

Purpose: Create isolated session workspace and discover available models dynamically.

Why Session-Based Workspaces:

Using a fixed directory like ai-docs/reviews/ causes problems:

• ❌ Multiple sessions overwrite each other's files
• ❌ Stale data from previous sessions pollutes results
• ❌ Hard to track which files belong to which session

Instead, create a unique session directory for each validation:

# Generate unique session ID
TARGET_SLUG=$(echo "${TASK_NAME:-review}" | tr '[:upper:] ' '[:lower:]-' | sed 's/[^a-z0-9-]//g' | head -c20)
SESSION_ID="review-${TARGET_SLUG}-$(date +%Y%m%d-%H%M%S)-$(head -c 4 /dev/urandom | xxd -p)"
SESSION_DIR="ai-docs/sessions/${SESSION_ID}"

# Create session workspace
mkdir -p "$SESSION_DIR"

echo "Session: $SESSION_ID"
echo "Directory: $SESSION_DIR"

# Example output:
# Session: review-auth-impl-20251212-143052-a3f2
# Directory: ai-docs/sessions/review-auth-impl-20251212-143052-a3f2

Benefits:

• ✅ Each session is isolated (no cross-contamination)
• ✅ Traceable - can associate files with a specific session
• ✅ Session ID can be used for tracking in statistics
• ✅ Parallel sessions don't conflict
• ✅ Aligned with the dev:dev session pattern
• ✅ Committed to git for audit trail (unlike /tmp/)

⚠️ Do NOT use /tmp/ for session directories. Files in /tmp/ are not traceable, not committable, and parallel runs will overwrite each other.

Dynamic Model Discovery:

NEVER hardcode model lists. Models change frequently — new ones appear, old ones deprecate, pricing updates. Instead, read the live catalog (list_models) for current available models:

Call the list_models MCP tool (claudish). It returns the current recommended set — model IDs, pricing, context window, capabilities, and the provider@model access prefixes — served from claudish’s catalog with a 24-hour cache.

For every live variant in one family, call search_models with the family name.

Recommended Free Models for Code Review:

Model Selection Flow (Learn and Reuse):

1. Read Preferences File
   → cat .claude/multimodel-team.json
   → If file NOT exists → create empty one

2. Detect Task Context
   → Parse task for keywords (case-insensitive):
     - "debug", "error", "bug", "fix", "trace", "issue" → debug
     - "research", "investigate", "analyze", "explore", "find" → research
     - "implement", "build", "create", "code", "develop", "feature" → coding
     - "review", "audit", "check", "validate", "verify" → review
   → If no keywords match → context = "default"

3. Check for Override Triggers in User Message
   → "use different models", "change models", "update preferences"
   → If found → force_ask = true

4. Load or Learn Models
   → models = contextPreferences[context]

   IF models exist AND NOT force_ask:
     → USE models directly (no asking)
     → Go to step 6

   IF models empty OR force_ask:
     → Read: the live catalog (list_models)
     → AskUserQuestion with multiSelect
     → Save user selection to contextPreferences[context]
     → Go to step 6

5. Save Updated Preferences
   → Write .claude/multimodel-team.json
   → Update lastUpdated timestamp

6. Execute with Models
   → Launch parallel validation
   → No further confirmation needed

Context Keywords:

Override Triggers (force re-selection):

• "use different models"
• "change models"
• "update model preferences"
• "select new models"

Routing is Claudish's

Send the id from list_models. Never build an address. Claudish owns backend selection, credentials and fallback; this repo implements none of it.

A prefix/backend/key table used to sit here. It is deleted — it had drifted to the wrong separator (/ where claudish uses @), listed alias env vars as if canonical, covered a third of the providers, and marked models "collision-free" that had since gained a direct provider. A second copy in claudish-usage had drifted differently, which is the point: restating claudish's routing anywhere in this repo guarantees two versions of the truth and no way to tell which is stale.

If a model will not route, that is a claudish bug — report it with report_error. Do not work around it by choosing a different prefix here.

Interactive Model Selection (AskUserQuestion with multiSelect):

CRITICAL: Use AskUserQuestion tool with multiSelect: true to let users choose models interactively. This provides a better UX than just showing recommendations.

// Use AskUserQuestion to let user select models
AskUserQuestion({
  questions: [{
    question: "Which external models should validate your code? (Internal Claude reviewer always included)",
    header: "Models",
    multiSelect: true,
    options: [
      // Top paid (from the live catalog (list_models) + historical data)
      {
        label: "grok ⚡",
        description: "$0.85/1M | Quality: 87% | Avg: 42s | Fast + accurate"
      },
      {
        label: "gemini",
        description: "$7.00/1M | Quality: 91% | Avg: 55s | High accuracy"
      },
      // Free models — filter the list_models result by pricing
      {
        label: "qwen/LATEST_FREE_CODING_MODEL 🆓",
        description: "FREE | Quality: 82% | 262K context | Coding-specialized"
      },
      {
        label: "mistralai/LATEST_FREE_CODING_MODEL 🆓",
        description: "FREE | 262K context | Dev-focused, new model"
      }
    ]
  }]
})

Remember Selection for Session:

Store the user's model selection in the session directory so it persists throughout the validation:

# After user selects models, save to session
save_session_models() {
  local session_dir="$1"
  shift
  local models=("$@")

  # Always include internal reviewer
  echo "claude-embedded" > "$session_dir/selected-models.txt"

  # Add user-selected models
  for model in "${models[@]}"; do
    echo "$model" >> "$session_dir/selected-models.txt"
  done

  echo "Session models saved to $session_dir/selected-models.txt"
}

# Load session models for subsequent operations
load_session_models() {
  local session_dir="$1"
  cat "$session_dir/selected-models.txt"
}

# Usage:
# After AskUserQuestion returns selected models
save_session_models "$SESSION_DIR" "grok" "qwen/LATEST_FREE_CODING_MODEL"

# Later in the session, retrieve the selection
MODELS=$(load_session_models "$SESSION_DIR")

Session Model Memory Structure:

$SESSION_DIR/
├── selected-models.txt    # User's model selection (persists for session)
├── claude-review.md       # Internal review
├── grok-review.md         # External review (if selected)
├── qwen-coder-review.md   # External review (if selected)
└── consolidated-review.md # Final consolidated review

Why Remember the Selection:

1. Re-runs: If validation needs to be re-run, use same models
2. Consistency: All phases of validation use identical model set
3. Audit trail: Know which models produced which results
4. Cost tracking: Accurate cost attribution per session

Always Include Internal Reviewer:

BEST PRACTICE: Always run internal Claude reviewer alongside external models.

Why?
✓ FREE (embedded Claude, no API costs)
✓ Fast baseline (usually fastest)
✓ Provides comparison point
✓ Works even if ALL external models fail
✓ Consistent behavior (same model every time)

The internal reviewer should NEVER be optional - it's your safety net.

In the code-review panels `dev` dispatches — where claudish is optional — it is not a
claudish slot. Launch it as its own
`Agent(subagent_type: "dev:reviewer", run_in_background: false, …)` in the SAME message
as the `team` call, carrying the contract lines — `TARGET: BRANCH`, `FOCUS:`, its own
`OUTPUT:` path, and `MODELS:` naming the externals. Never seat it in that team's `models`
list. Its return is in your hands: list its OUTPUT file on REVIEWS: whether or not it
carries a `**Verdict**:` line — the synthesizer counts a file with no verdict as
no-verdict, never as one more approval. (`/team` itself, where claudish is a hard
dependency, seats it as the `internal` slot instead — Pattern 3 states the rule.)

Pattern 1: The 4-Message Pattern (MANDATORY)

This pattern is CRITICAL for achieving true parallel execution with multiple AI models.

Why This Pattern Exists:

Claude Code executes tools sequentially by default when different tool types are mixed in the same message. To achieve true parallelism, you MUST:

1. Use ONLY one tool type per message
2. Ensure all Agent calls are in a single message
3. Separate preparation (Bash) from execution (Task) from presentation

The Pattern:

Message 1: Preparation (Bash Only)
  - Create workspace directories
  - Validate inputs (check if claudish installed)
  - Write the brief (input.md) — never a pre-computed diff: every reviewer is
    handed `TARGET: BRANCH` and captures its own surfaces through dev's
    `capture-review-surfaces.ts`
  - NO Agent calls
  - NO Tasks calls

Message 2: Parallel Execution (the internal Agent call and ONE team call, same message)
  - `Agent(subagent_type: "dev:reviewer", run_in_background: false, …)` — the
    internal reviewer, always present, carrying the contract lines
  - Every external model in a single `team` MCP call; the tool parallelises them
    internally. In a dev-dispatched panel the internal reviewer is never a
    `models` entry (Pattern 3 — `/team` itself seats it as the `internal` slot)
  - Pass require_pattern whenever the prompt mandates an output shape
  - (A pure-Agent fan-out with no external models still obeys the
     one-tool-type-per-message rule above)

Message 3: Auto-Consolidation (Task Only)
  - Automatically triggered when the panel settles — at N = 1 too, where the
    synthesizer passes the single review through with a `VERDICT:` line (Pattern 5)
  - Launch `dev:synthesizer` — the only consolidator; it reads reviews, never code
  - Pass every review file path on REVIEWS:, the three lines under 'Apply verdict
    thresholds' in `dev:reviewer`'s agent file on THRESHOLDS: — read at dispatch
    time, never recalled — and the consolidated file on OUTPUT: (Pattern 5)
  - Consensus analysis is the synthesizer's; never send the reviews to dev:reviewer

Message 4: Present Results
  - Show user prioritized issues
  - Include consensus levels (unanimous, strong, majority)
  - Link to detailed reports
  - Cost summary (if applicable)

Example: 5-Model Parallel Code Review

Message 1: Preparation (Session Setup + Model Discovery)
  # Create unique session workspace
  Bash: SESSION_ID="review-$(date +%Y%m%d-%H%M%S)-$(head -c 4 /dev/urandom | xxd -p)"
  Bash: SESSION_DIR="ai-docs/sessions/${SESSION_ID}" && mkdir -p "$SESSION_DIR"
  # No code capture here. Every reviewer is handed TARGET: BRANCH and runs dev's
  # capture-review-surfaces.ts itself, in BRANCH mode. A range computed here would
  # be one more hand-rolled diff, which is the one thing no dispatcher may do.

  # Discover available models
  MCP:  list_models   # current models, pricing, capabilities

  # User selects models via AskUserQuestion (see Pattern 0)

Message 2: Start the panel (the internal Agent call and ONE team call, same message)
  Bash: write the brief to "$SESSION_DIR/input.md" — the contract lines every
        external gets: TARGET: BRANCH / FOCUS: code / MODELS: none

  Agent(
    subagent_type: "dev:reviewer",
    run_in_background: false,
    description: "Internal code review",
    prompt: "TARGET: BRANCH
             FOCUS: code
             OUTPUT: $SESSION_DIR/claude-review.md
             MODELS: grok,LATEST_FREE_CODING_MODEL,gpt,LATEST_FREE_REASONING_MODEL"
  )
  ---
  claudish team(mode="run", path=$SESSION_DIR,
    models=["grok", "LATEST_FREE_CODING_MODEL", "gpt", "LATEST_FREE_REASONING_MODEL"],
    input_file="$SESSION_DIR/input.md",
    require_pattern="\*\*Verdict\*\*: (PASS|CONDITIONAL|FAIL)", agent="dev:reviewer")

  All 5 reviewers run at once: the Agent is the always-present internal reviewer,
  and the team tool parallelises the four externals internally. The internal
  reviewer is never a `models` entry — its return is in your hands, and its OUTPUT
  file goes on REVIEWS: whether or not it carries a `**Verdict**:` line; the
  synthesizer counts a file with no verdict as no-verdict, never as approval.

  The team call RETURNS IMMEDIATELY with a slots map. It does not carry the reviews.

Message 2b: Poll to completion
  claudish team(mode="status", path=$SESSION_DIR)
  # repeat until no slot in `models` has state === "RUNNING"
  # bound the loop; read idle_seconds_by_slot with activity_by_slot before
  # concluding a quiet slot is stuck

Message 3: Auto-Consolidation
  (Automatically triggered - don't wait for user to request)

  # The internal reviewer wrote $SESSION_DIR/claude-review.md. The team run wrote
  # one file per external slot, $SESSION_DIR/response-NN.md, named by
  # ANONYMOUS slot id rather than by model, because the vote is blind. Do not try
  # to attribute a file to a model before the verdict is in. The synthesizer is
  # the only consolidator: it is given the reviews and never the code, and no
  # reviewer ever sees another reviewer's output.

  Agent(
    subagent_type: "dev:synthesizer",
    run_in_background: false,
    description: "Consolidate code reviews",
    prompt: "REVIEWS: $SESSION_DIR/claude-review.md
             $SESSION_DIR/response-01.md
             $SESSION_DIR/response-02.md
             $SESSION_DIR/response-03.md
             $SESSION_DIR/response-04.md
             THRESHOLDS: <the three lines under 'Apply verdict thresholds' in
                          dev:reviewer's agent file, read at dispatch time, never
                          recalled>
             OUTPUT: $SESSION_DIR/consolidated-review.md
             Consolidate with consensus levels (unanimous / strong / majority / divergent).
             Compute the verdict line from your counts against THRESHOLDS.
             You are given reviews, never code. Do not review."
  )

Message 4: Present Results + Update Statistics
  # Track performance for each model (see Pattern 7)
  track_model_performance "claude-embedded" "success" 32 8 95
  track_model_performance "grok" "success" 45 6 87
  track_model_performance "qwen/LATEST_FREE_CODING_MODEL" "success" 52 5 82
  track_model_performance "gpt" "success" 68 7 89
  track_model_performance "mistralai/LATEST_FREE_CODING_MODEL" "success" 48 5 84

  # Record session summary
  record_session_stats 5 5 0 68 245 3.6

  "Multi-model code review complete! 5 AI models analyzed your code.
   Session: $SESSION_ID

   Top 5 Issues (Prioritized by Consensus):
   1. [UNANIMOUS] Missing input validation on POST /api/users
   2. [UNANIMOUS] SQL injection risk in search endpoint
   3. [STRONG] Weak password hashing (bcrypt rounds too low)
   4. [MAJORITY] Missing rate limiting on authentication endpoints
   5. [MAJORITY] Insufficient error handling in payment flow

   Model Performance (this session):
   | Model                          | Time | Issues | Quality | Cost   |
   |--------------------------------|------|--------|---------|--------|
   | claude-embedded                | 32s  | 8      | 95%     | FREE   |
   | grok          | 45s  | 6      | 87%     | $0.002 |
   | qwen/LATEST_FREE_CODING_MODEL          | 52s  | 5      | 82%     | FREE   |
   | gpt        | 68s  | 7      | 89%     | $0.015 |
   | mistralai/LATEST_FREE_CODING_MODEL   | 48s  | 5      | 84%     | FREE   |

   Parallel Speedup: 3.6x (245s sequential → 68s parallel)

   See $SESSION_DIR/consolidated-review.md for complete analysis.
   Performance logged to ai-docs/llm-performance.json"

Performance Impact:

• Sequential execution: 5 models × 3 min = 15 minutes
• Parallel execution: max(model times) ≈ 5 minutes
• Speedup: 3x with perfect parallelism

Pattern 2: Parallel Execution Architecture

Single Message, Multiple Tasks:

The key to parallel execution is putting ALL Agent calls in a single message with the --- delimiter:

✅ CORRECT - Parallel Execution:

Agent: <agent-1>
  Prompt: "Task 1 instructions"
---
Agent: <agent-2>
  Prompt: "Task 2 instructions"
---
Agent: <agent-3>
  Prompt: "Task 3 instructions"

All 3 execute simultaneously.

Anti-Pattern: Sequential Execution

❌ WRONG - Sequential Execution:

Message 1:
  Agent: <agent-1>
Message 2:
  Agent: <agent-2>
Message 3:
  Agent: <agent-3>
Each task waits for previous to complete (3x slower).

Independent Tasks Requirement:

Each Task must be independent (no dependencies):

✅ CORRECT - Independent:
  Task: review code for security
  Task: review code for performance
  Task: review code for style

  All can run simultaneously (same input, different perspectives).

❌ WRONG - Dependent:
  Task: implement feature
  Task: write tests for feature (depends on implementation)
  Task: review implementation (depends on tests)

  Must run sequentially (each needs previous output).

Unique Output Files:

Each Task MUST write to a unique output file within the session directory:

✅ CORRECT - Unique Files in Session Directory:
  Task: reviewer1 → $SESSION_DIR/claude-review.md
  Task: reviewer2 → $SESSION_DIR/grok-review.md
  Task: reviewer3 → $SESSION_DIR/qwen-coder-review.md

❌ WRONG - Shared File:
  Task: reviewer1 → $SESSION_DIR/review.md
  Task: reviewer2 → $SESSION_DIR/review.md (overwrites reviewer1!)
  Task: reviewer3 → $SESSION_DIR/review.md (overwrites reviewer2!)

❌ WRONG - Fixed Directory (not session-based):
  Task: reviewer1 → ai-docs/reviews/claude-review.md  # May conflict with other sessions!

Wait for All Before Consolidation:

Do NOT consolidate until ALL tasks complete:

✅ CORRECT - Wait for All:
  Launch: Task1, Task2, Task3, Task4 (parallel)
  Wait: All 4 complete
  Check: results.filter(r => r.status === 'fulfilled').length
  If >= 1: Dispatch dev:synthesizer (a passthrough with a verdict at N = 1);
           if any failed, also offer to retry them
  If 0:    Offer retry or abort

❌ WRONG - Premature Consolidation:
  Launch: Task1, Task2, Task3, Task4
  After 30s: Task1, Task2 done
  Consolidate: Only Task1 + Task2 (Task3, Task4 still running!)

Pattern 3: Model Invocation via claudish MCP

How models are invoked:

External models are invoked via claudish MCP tools. The orchestrator calls the MCP tools directly; no Bash invocation is needed. This is 100% reliable.

Where the native reviewer sits — one rule. It is a claudish internal slot when claudish is a hard dependency of the dispatcher, and a separate Agent(…) call when claudish is optional.

• /team itself — the multimodel plugin declares claudish as a dependency, so internal goes in the models list of the ONE team call and gets the same require_pattern shape check as every external. That procedure lives in this plugin's own commands/team.md (Step 2 onward) and is not restated here.
• The code-review panels dev dispatches (/dev:dev Phase 5, /dev:audit, /dev:fix Phase B) — claudish is optional there and the panel may be empty, so the internal reviewer is an always-present Agent(subagent_type: "dev:reviewer", run_in_background: false, …) issued in the same message as the team call, carrying the contract lines. Every code-review example in this skill is this case.

For a dev-dispatched code-review panel: write the brief to input.md, start the internal Agent and the panel in one message, poll the panel to completion, then read the externals' reviews off disk.

Agent(subagent_type: "dev:reviewer", run_in_background: false,
  description: "Internal code review",
  prompt: "TARGET: BRANCH\nFOCUS: code\nOUTPUT: ${SESSION_DIR}/claude-review.md\nMODELS: grok,gemini")
---
team(mode="run", path=SESSION_DIR, models=["grok", "gemini"],
  input_file=`${SESSION_DIR}/input.md`,
  require_pattern="\*\*Verdict\*\*: (PASS|CONDITIONAL|FAIL)", agent="dev:reviewer")
  // → { started: true, slots: { "grok": "01", "gemini": "02" }, ... }

team(mode="status", path=SESSION_DIR)   // until no slot has state === "RUNNING"
  // → read `${SESSION_DIR}/response-<slot>.md` for each slot

The team tool runs all models in parallel internally. run does not wait and does not return the answers — it starts the slots and hands back the slot map. Per-model status (state, exitCode, outputSize, error.reason) comes from a settled status response. There is no timeout parameter any more, and passing one is silently ignored. Full procedure: claudish:claudish-usage → "The three-step lifecycle". Requires claudish >= 8.0.0.

In a dev-dispatched panel the internal reviewer is never a models entry. It runs as the Agent above — on the host session, with the dev plugin loaded, so TARGET: BRANCH resolves through dev's own capture script — and its return is in the dispatcher's hands. Its file goes on REVIEWS: whether or not it carries a verdict line; the synthesizer counts a file with no verdict as no-verdict, never as one more approval.

require_pattern is what turns exit 0 into a real success check. A slot that finished without producing the required shape is reported FAILED (state EMPTY, reason shape_mismatch) instead of counted as a success. Exit code 0 also occurs on API errors and on a child that simply ignored the format, so without this the panel can report a verdict it never actually received.

For single-model delegation:

create_session(model="grok", prompt=TASK_PROMPT, timeout_seconds=300)
→ channel events: session_started → tool_executing → completed/failed
→ get_output(session_id) to retrieve result

Verification:

• team tool: check status.models[<slot>].state on a SETTLED status response — never the run response, which returns before any model has answered
• create_session: The completed channel event confirms success; failed provides error details

Correct Pattern Example

// ✅ CORRECT (dev-dispatched panel): the internal reviewer as its own Agent, every external in ONE team call — same message
Agent({ subagent_type: "dev:reviewer", run_in_background: false,
        description: "Internal code review",
        prompt: "TARGET: BRANCH\nFOCUS: code\nOUTPUT: ${SESSION_DIR}/claude-review.md\nMODELS: grok,gemini" })
team(mode="run", path=SESSION_DIR,
  models=["grok", "gemini"],
  input_file=`${SESSION_DIR}/input.md`,
  require_pattern="\*\*Verdict\*\*: (PASS|CONDITIONAL|FAIL)", agent="dev:reviewer")

// …then poll, then read the externals' reviews from response-<slot>.md
team(mode="status", path=SESSION_DIR)

// ❌ WRONG: the native reviewer fired into the background, its file read by nobody
Agent({ subagent_type: "dev:reviewer", run_in_background: true,
        prompt: "Review the change...\n\nWrite to: session/internal-result.md" })

The internal reviewer runs run_in_background: false so that its return — and whether its OUTPUT file carries a **Verdict**: line — is in hand before the synthesizer is dispatched. It goes on REVIEWS: either way; the synthesizer counts a file with no verdict as no-verdict, never as approval. A background Agent's file is read by nobody until it is too late. (For /team itself the correct form is the internal slot — commands/team.md, per the rule above.)

Pattern 4: Cost Estimation and Transparency

Input/Output Token Separation:

Provide separate estimates for input and output tokens:

Cost Estimation for Multi-Model Review:

Input Tokens (per model):
  - Code context: 500 lines × 1.5 = 750 tokens
  - Review instructions: 200 tokens
  - Total input per model: ~1000 tokens
  - Total input (5 models): 5,000 tokens

Output Tokens (per model):
  - Expected output: 2,000 - 4,000 tokens
  - Total output (5 models): 10,000 - 20,000 tokens

Cost Calculation (example rates):
  - Input: 5,000 tokens × $0.0001/1k = $0.0005
  - Output: 15,000 tokens × $0.0005/1k = $0.0075 (3-5x more expensive)
  - Total: $0.0080 (range: $0.0055 - $0.0105)

User Approval Gate:
  "Multi-model review will cost approximately $0.008 ($0.005 - $0.010).
   Proceed? (Yes/No)"

Input Token Estimation Formula:

Input Tokens = (Code Lines × 1.5) + Instruction Tokens

Why 1.5x multiplier?
  - Code lines: ~1 token per line (average)
  - Context overhead: +50% (imports, comments, whitespace)

Example:
  500 lines of code → 500 × 1.5 = 750 tokens
  + 200 instruction tokens = 950 tokens total input

Output Token Estimation Formula:

Output Tokens = Base Estimate + Complexity Factor

Base Estimates by Task Type:
  - Code review: 2,000 - 4,000 tokens
  - Design validation: 1,000 - 2,000 tokens
  - Architecture planning: 3,000 - 6,000 tokens
  - Bug investigation: 2,000 - 5,000 tokens

Complexity Factors:
  - Simple (< 100 lines code): Use low end of range
  - Medium (100-500 lines): Use mid-range
  - Complex (> 500 lines): Use high end of range

Example:
  400 lines of complex code → 4,000 tokens (high complexity)
  50 lines of simple code → 2,000 tokens (low complexity)

Range-Based Estimates:

Always provide a range (min-max), not a single number:

✅ CORRECT - Range:
  "Estimated cost: $0.005 - $0.010 (depends on review depth)"

❌ WRONG - Single Number:
  "Estimated cost: $0.0075"
  (User surprised when actual is $0.0095)

Why Output Costs More:

Output tokens are typically 3-5x more expensive than input tokens:

Example Pricing (OpenRouter):
  - Grok: $0.50 / 1M input, $1.50 / 1M output (3x difference)
  - Gemini Flash: $0.10 / 1M input, $0.40 / 1M output (4x difference)
  - GPT-5 Codex: $1.00 / 1M input, $5.00 / 1M output (5x difference)

Impact:
  If input = 5,000 tokens, output = 15,000 tokens:
    Input cost: $0.0005
    Output cost: $0.0075 (15x higher despite only 3x more tokens)
    Total: $0.0080 (94% is output!)

User Approval Before Execution:

ALWAYS ask for user approval before expensive operations:

Present to user:
  "You selected 5 AI models for code review:
   - Claude Sonnet (embedded, free)
   - Grok Code Fast (external, $0.002)
   - Gemini 2.5 Flash (external, $0.001)
   - GPT-5 Codex (external, $0.004)
   - DeepSeek Coder (external, $0.001)

   Estimated total cost: $0.008 ($0.005 - $0.010)

   Proceed with multi-model review? (Yes/No)"

If user says NO:
  Offer alternatives:
    1. Use only free embedded Claude
    2. Select fewer models
    3. Cancel review

If user says YES:
  Proceed with Message 2 (parallel execution)

Pattern 5: Auto-Consolidation Logic

Automatic Trigger:

Consolidation happens automatically as soon as the panel settles — at N = 1 as well as N ≥ 2. dev:synthesizer is the only writer of the consolidated report; at N = 1 it passes the single review through unchanged and appends the VERDICT: line, so the output has one shape whatever N is:

✅ CORRECT - Auto-Trigger:

const results = await Promise.allSettled([task1, task2, task3, task4, task5]);
const successful = results.filter(r => r.status === 'fulfilled');
const failed = results.length - successful.length;

if (successful.length >= 1) {
  // Auto-trigger consolidation (DON'T wait for user to ask). N = 1 is a passthrough with a verdict.
  const reviewPaths = successful.map((r) => r.value.reviewFile); // one review file per slot
  const consolidated = await Agent({
    subagent_type: "dev:synthesizer",
    run_in_background: false,     // formatResults() consumes the return value
    description: "Consolidate code reviews",
    prompt: `REVIEWS: ${reviewPaths.join("\n")}
THRESHOLDS: <the three lines under 'Apply verdict thresholds' in dev:reviewer's agent file, read at dispatch time, never recalled>
OUTPUT: ${SESSION_DIR}/consolidated-review.md
Consolidate with consensus levels (unanimous / strong / majority / divergent).
Compute the verdict line from your counts against THRESHOLDS.
You are given reviews, never code. Do not review.`
  });

  if (failed > 0) {
    // An addition to the dispatch above, never a substitute for it
    notifyUser(`${failed} of ${results.length} models failed. Retry them and re-consolidate?`);
  }
  return formatResults(consolidated);
} else {
  // All failed — there is nothing to pass through
  notifyUser("All models failed. Check logs and retry?");
}

❌ WRONG - Wait for User:

const results = await Promise.allSettled([...]);
const successful = results.filter(r => r.status === 'fulfilled');

// Present results to user
notifyUser("3 reviews complete. Would you like me to consolidate them?");
// Waits for user to request consolidation...

❌ WRONG - Skip the synthesizer at N = 1:

if (successful.length >= 2) {
  await consolidate();
} else {
  notifyUser("Only 1 model succeeded. See single review or retry?");
  // The raw review carries no VERDICT: line and not the shape every other dispatcher reads
}

Why Auto-Trigger:

• Better UX (no extra user prompt needed)
• Faster workflow (no wait for user response)
• Expected behavior (user assumes consolidation is part of workflow)

N = 1 is a passthrough, not a skip:

Consensus levels need at least two reviews; the consolidated report does not. At N = 1 the synthesizer emits the single review unchanged — no [CONSENSUS: …] tags, nothing reworded — followed by the VERDICT: line computed from that review's own counts against THRESHOLDS, so the dispatcher still gets the one file its gate reads. The only dispatcher that skips the synthesizer at N = 1 is /dev:fix Phase B, whose output is a vote tally, and a single vote is its own tally:

if (successful.length >= 1) {
  // Dispatch dev:synthesizer: consolidation at N ≥ 2, passthrough with a verdict at N = 1
  if (successful.length < results.length) {
    // In addition, not instead
    notifyUser("Some models failed. Retry the failures and re-consolidate?");
  }
} else {
  // All failed
  notifyUser("All models failed. Check logs and retry?");
}

Pass All Review File Paths:

dev:synthesizer needs the path of EVERY review file, one per REVIEWS: line. It reads the reviews and never the code, so the paths are all it gets:

Agent(
  subagent_type: "dev:synthesizer",
  run_in_background: false,
  description: "Consolidate code reviews",
  prompt: "REVIEWS: $SESSION_DIR/claude-review.md
           $SESSION_DIR/grok-review.md
           $SESSION_DIR/qwen-coder-review.md
           THRESHOLDS: <the three lines under 'Apply verdict thresholds' in
                        dev:reviewer's agent file, read at dispatch time, never
                        recalled>
           OUTPUT: $SESSION_DIR/consolidated-review.md
           Consolidate with consensus levels (unanimous / strong / majority / divergent).
           Compute the verdict line from your counts against THRESHOLDS.
           You are given reviews, never code. Do not review."
)

Don't Inline Full Reviews:

❌ WRONG - Inline Reviews (context pollution):
  Prompt: "Consolidate these reviews:

           Claude Review:
           [500 lines of review content]

           Grok Review:
           [500 lines of review content]

           Qwen Review:
           [500 lines of review content]"

✅ CORRECT - File Paths in Session Directory:
  prompt: "REVIEWS: $SESSION_DIR/claude-review.md
           $SESSION_DIR/grok-review.md
           $SESSION_DIR/qwen-coder-review.md
           THRESHOLDS: ...
           OUTPUT: $SESSION_DIR/consolidated-review.md
           ..."

Pattern 6: Consensus Analysis

Consensus Levels:

Classify issues by how many models flagged them:

Consensus Levels (for N models):

UNANIMOUS (100% agreement):
  - All N models flagged this issue
  - VERY HIGH confidence
  - MUST FIX priority

STRONG CONSENSUS (67-99% agreement):
  - Most models flagged this issue (⌈2N/3⌉ to N-1)
  - HIGH confidence
  - RECOMMENDED priority

MAJORITY (50-66% agreement):
  - Half or more models flagged this issue (⌈N/2⌉ to ⌈2N/3⌉-1)
  - MEDIUM confidence
  - CONSIDER priority

DIVERGENT (< 50% agreement):
  - Only 1-2 models flagged this issue
  - LOW confidence
  - OPTIONAL priority (may be model-specific perspective)

Example: 5 Models

Issue Flagged By:              Consensus Level:    Priority:
─────────────────────────────────────────────────────────────
All 5 models                   UNANIMOUS (100%)    MUST FIX
4 models                       STRONG (80%)        RECOMMENDED
3 models                       MAJORITY (60%)      CONSIDER
2 models                       DIVERGENT (40%)     OPTIONAL
1 model                        DIVERGENT (20%)     OPTIONAL

Keyword-Based Matching (v1.0):

Simple consensus analysis using keyword matching:

Algorithm:

1. Extract issues from each review
2. For each unique issue:
   a. Identify keywords (e.g., "SQL injection", "input validation")
   b. Check which other reviews mention same keywords
   c. Count models that flagged this issue
   d. Assign consensus level

Example:

Claude Review: "Missing input validation on POST /api/users"
Grok Review: "Input validation absent in user creation endpoint"
Gemini Review: "No validation for user POST endpoint"

Keywords: ["input validation", "POST", "/api/users", "user"]
Match: All 3 reviews mention these keywords
Consensus: UNANIMOUS (3/3 = 100%)

Model Agreement Matrix:

Show which models agree on which issues:

Issue Matrix:

Issue                             Claude  Grok  Gemini  GPT-5  DeepSeek  Consensus
──────────────────────────────────────────────────────────────────────────────────
SQL injection in search              ✓      ✓     ✓       ✓       ✓      UNANIMOUS
Missing input validation

…(truncated)