rohitg00 LLM Council
Karpathy's LLM Council pattern, provider-agnostic. dair-academy's version hardcoded Fireworks; ours reads any OpenAI-compatible endpoint via env.
When to use
- High-stakes plan review (
/plancrosses N-file threshold) - Conflicting learning-rules → re-resolve via vote
- User invokes
/council "<query>"or/wiki council - Architecture decisions where you want multiple viewpoints captured
- Persisting deliberation as a wiki page for future reference
Three phases
- Independent: each model answers in parallel
- Ranking: each model ranks anonymized peer responses
- Synthesis: chairman model reads all responses + rankings → final answer
Provider config
Provider chosen via env. First-match wins:
| Env var | Provider | Default base URL |
|---|---|---|
ANTHROPIC_API_KEY |
Anthropic | https://api.anthropic.com |
OPENAI_API_KEY |
OpenAI | https://api.openai.com/v1 |
OPENROUTER_API_KEY |
OpenRouter | https://openrouter.ai/api/v1 |
FIREWORKS_API_KEY |
Fireworks | https://api.fireworks.ai/inference/v1 |
LLM_COUNCIL_BASE_URL + LLM_COUNCIL_API_KEY |
Custom OpenAI-compat | (user-supplied) |
Override per-run with --provider openai|anthropic|openrouter|fireworks|custom.
Default model rosters per provider live in scripts/council.js and can be overridden via --models CSV and --chairman <id>.
Commands
node $SKILL_ROOT/scripts/council.js run "<query>" [--models id1,id2,id3] [--chairman id] [--provider <name>] [--wiki <slug>]
node $SKILL_ROOT/scripts/council.js providers
node $SKILL_ROOT/scripts/council.js show <session-id>
--wiki <slug> writes the full transcript to <wiki>/derived/council/<session-id>.md and registers it via wiki-cli.js page so it shows in FTS5 search.
Output
Each session writes:
~/.pro-workflow/council/<session-id>/
├── config.json # query, models, chairman, provider
├── phase1_responses.json # raw API responses per model
├── phase2_rankings.json # anonymized ranking outputs
├── phase3_synthesis.txt # chairman's final answer
└── final_output.md # human-readable bundle
Console prints the markdown bundle. Pipe to pbcopy / tee as needed.
Hard rules
- Never skip the ranking phase. It's the core of the council pattern.
- Save raw responses to disk verbatim. No summarization in storage.
- Anonymize responses for ranking — models see
Response A/B/C/..., not peer names. - The chairman sees both real names AND rankings.
- Display all three phases to the user. No phase elision.
Cost awareness
The script logs per-call latency + tokens on supported providers. Multiply by your provider rate to estimate. Council cost grows linearly with len(models)^2 (each model ranks all others) plus the chairman.
Default council size: 3-5 models. More models = exponentially more ranking calls.
Use with wiki
/wiki council agent-memory "should we adopt episodic memory in our agents?"
Loads agent-memory wiki context as system prompt prefix, runs council, persists transcript as wiki/derived/council/<id>.md. The transcript becomes searchable via /wiki ask.