LLM Council

LLM Council

You ask one AI a question, you get one answer. That answer might be great. It might be mid. You have no way to tell, because you only saw one perspective.

The council fixes this. Five independent advisors analyze the question from fundamentally different angles, peer-review each other anonymously, and a chairman synthesizes a final verdict that tells the user where the advisors agree, where they clash, and what to actually do.

Adapted from Andrej Karpathy's LLM Council. He dispatches queries to multiple models, has them peer-review each other anonymously, then a chairman produces the final answer. This does the same thing with Claude subagents — distinct thinking lenses and fully isolated contexts instead of different vendors. No API keys. Every advisor is normal subagent usage on the user's own subscription.

When to run the council

The council is for questions where being wrong is expensive.

Good council questions:

• "Should I launch a $97 workshop or a $497 course?"
• "Which of these 3 positioning angles is strongest?"
• "I'm thinking of pivoting from X to Y. Am I crazy?"
• "Here's my landing page copy. What's weak?"
• "Should I hire a VA or build an automation first?"

Bad council questions:

• "What's the capital of France?" (one right answer)
• "Write me a tweet" (creation task, not a decision)
• "Summarize this article" (processing task, not judgment)

The council shines when there's genuine uncertainty and the cost of a bad call is high. If the user already knows the answer and just wants validation, the council will likely tell them things they don't want to hear. That's the point.

The five advisors

Not job titles — thinking styles that create tension with each other.

Why these five: they create three natural tensions. Contrarian vs Expansionist (downside vs upside). First Principles vs Executor (rethink everything vs just do it). The Outsider sits in the middle keeping everyone honest.

Why the Outsider gets no context: it is the only advisor that catches what's obvious to the user and confusing to everyone else. Enriched context is exactly what destroys that. It receives the bare question, reads no files, searches nothing. This is deliberate — do not "help" it.

Stage 1 — Frame the question

A. Gather context (30 seconds, not more). Read CLAUDE.md if present and any files the user referenced. You're looking for the 2-3 things that let advisors give specific, grounded advice instead of generic takes: business stage, audience, constraints, past results, relevant numbers.

B. Write the context blurb. 1-3 sentences plus the paths worth investigating. Advisors run in the working directory and will read those paths themselves — something an external-API council can never do.

C. Pass the question verbatim. Don't add your opinion, don't steer it. Framing goes in context, not in the question.

If the question is too vague ("council this: my business"), ask one clarifying question. Just one. Then proceed.

Stage 2 — Convene

Invoke the Workflow tool with the bundled script (runs in background; you'll be notified when it's done). council-workflow.js lives next to this SKILL.md — resolve its absolute path from wherever the skill is installed (user-level: ~/.claude/skills/llm-council/council-workflow.js; project-level: <project>/.claude/skills/llm-council/council-workflow.js).

Workflow({
  scriptPath: "<absolute path to council-workflow.js>",
  args: {
    question: "<the user's question, verbatim>",
    mode: "full",       // "full" (default) = with peer review; "quick" = opinions → synthesis
    context: "<1-3 sentences of workspace/session context, including paths worth reading>",
    members: [          // optional override (2-8) — see Custom councils below
      { persona: "...", brief: "...", model: "opus", context: "full" }
    ]
  }
})

Use mode: "quick" when the user says quick/cheap/fast, or the question deserves five lenses but not a full cross-critique round. Use full otherwise — the anonymized peer review is where most of the value comes from.

Custom councils. If the user asks for a custom lineup ("a council of security experts about X"), seat it via members. Be aware you're trading away the design: the default five are a tension system, and five experts in one field tend to agree with each other, which is the failure mode the council exists to prevent. Seat custom personas when asked, but build in disagreement — give at least one member an adversarial brief.

Stage 3 — Chairman synthesis (you, after the workflow returns)

The workflow returns { question, mode, council[], peer_reviews[], aggregate_ranking[] }. Each council entry has { id, persona, model, stance, confidence, key_points, answer }. Peer reviews arrive de-anonymized. aggregate_ranking gives each advisor's average peer rank (lower = better).

Present the verdict in chat as markdown. Do not generate files.

## Council Verdict: {short topic}

### Where the Council Agrees
{Points multiple advisors converged on independently. High-confidence signals.}

### Where the Council Clashes
{Genuine disagreements. Present both sides. Don't smooth them over — explain why reasonable advisors disagree.}

### Blind Spots the Council Caught
{Things that only emerged in peer review. Weight `missed_by_all` heavily — that's the thing nobody had.}

### Council Verdict Table
| Advisor | Stance | Avg peer rank |

### The Recommendation
{A clear, direct recommendation. Not "it depends." Not "consider both sides." A real answer with reasoning.}

### The One Thing to Do First
{A single concrete next step. Not a list of ten. One thing.}

Close with one line noting the council was N independent Claude agents with fresh contexts, and offering to publish the verdict as a shareable page if the user wants one.

Reading the rank table. Average rank measures how well each advisor executed its lens — specificity, evidence, whether it surfaced something real. It does not measure whose advice to follow. The Contrarian ranking low usually means it did its job. Say this if you cite the table.

The chairman can overrule the council. If four advisors say "do it" but the dissenter's reasoning is strongest, side with the dissenter and say why. Apply your own judgment — just be explicit when you're departing from the council.

Attribute inline. "The Outsider flagged…", "The Executor's path assumes…". The user should be able to see which lens produced which insight.

Transcripts

Only save a transcript if the user asks or the question is significant enough to reference later. Write council-transcript-[YYYY-MM-DD-HHMM].md to the working directory.

Fallback (Workflow tool unavailable)

Reproduce the stages with parallel Agent calls: spawn all five advisors in one message (one Agent call per advisor, prompts containing lens brief + question + ground rules, and the Outsider's prompt carrying the bare question with no context and an instruction not to explore). Then anonymize the answers as Response A-E, spawn five reviewers in one message with the anonymized packet and the four review questions, then synthesize as above. Compute average ranks yourself.

Error handling

• Advisors that fail are reported in advisors_failed. Proceed with who responded and tell the user which seats were empty.
• If the whole workflow fails, say so and offer a single-perspective analysis instead.

Integrity

Never present advisors as ChatGPT, Gemini, or any other vendor — they are Claude agents with distinct lenses and isolated contexts. If the user wants literally cross-vendor opinions, explain that this skill gives independent Claude perspectives, and true multi-vendor councils need API keys (karpathy/llm-council).