LLM Council

LLM Council Skill

LIBRARY-FIRST PROTOCOL (MANDATORY)

Before writing ANY code, you MUST check:

Step 1: Library Catalog

• Location: .claude/library/catalog.json
• If match >70%: REUSE or ADAPT

Step 2: Patterns Guide

• Location: .claude/docs/inventories/LIBRARY-PATTERNS-GUIDE.md
• If pattern exists: FOLLOW documented approach

Step 3: Existing Projects

• Location: D:\Projects\*
• If found: EXTRACT and adapt

Decision Matrix

Purpose

Run 3-stage multi-model consensus for critical decisions where:

• Single-model hallucination risk is unacceptable
• Multiple perspectives improve decision quality
• High-stakes choices need validation

Architecture (Karpathy Pattern)

STAGE 1: COLLECT
  +---> Claude ---> Response A
  |
Query --+---> Gemini ---> Response B
  |
  +---> Codex ----> Response C

STAGE 2: RANK
  Each model reviews others (anonymized)
  Produces rankings with rationale

STAGE 3: SYNTHESIZE
  Chairman aggregates rankings
  Produces final answer with consensus score

When to Use

Perfect For:

• Architecture decisions
• Technology selection
• Critical bug triage
• Security assessment
• High-risk deployments
• Contentious design choices

Don't Use When:

• Simple, low-risk decisions
• Time-critical responses
• Single correct answer exists
• Cost is a concern (3x API usage)

Usage

Basic Council

/llm-council "Should we use microservices or monolith for this system?"

With Threshold

/llm-council "Which auth approach is best?" --threshold 0.75

With Chairman Override

/llm-council "Architecture decision" --chairman gemini

Command Pattern

bash scripts/multi-model/llm-council.sh "<query>" "<threshold>" "<chairman>"

Configuration

Consensus Scoring

• >0.80: Strong consensus - proceed with confidence
• 0.67-0.80: Moderate consensus - consider minority views
• <0.67: Weak consensus - escalate to human review

Memory Integration

Results stored to Memory-MCP:

• Key: multi-model/council/decisions/{query_id}
• Tags: WHO=llm-council, WHY=consensus-decision

Output Format

{
  "query": "Original question",
  "final_answer": {
    "synthesis": "Combined answer...",
    "chairman": "claude"
  },
  "consensus_score": 0.85,
  "responses": {
    "claude": "...",
    "gemini": "...",
    "codex": "..."
  },
  "rankings": [
    {"model": "A", "rank": 1, "rationale": "..."}
  ]
}

Failure Modes

Deadlock (No Consensus)

• All models disagree
• Consensus < threshold
• Action: Store for human review

Model Unavailable

• One model times out
• Action: Continue with 2 models (2/3 quorum)

Chairman Failure

• Synthesis fails
• Action: Fallback to highest-ranked response

Integration Examples

Architecture Decision

const decision = await runCouncil(
  "Microservices vs Monolith for our scale?",
  { threshold: 0.75 }
);

if (decision.consensus_score >= 0.75) {
  proceed(decision.final_answer);
} else {
  escalateToHuman(decision);
}

Security Assessment

const assessment = await runCouncil(
  "Is this authentication approach secure?",
  { threshold: 0.80 }
);
// Higher threshold for security decisions

Sources

• LLM Council by Andrej Karpathy
• VentureBeat Analysis