LLM Arena
Version: 1.0.0 Author: Claude Opus 4.6 Date: 2026-02-12
Orchestrates multi-turn conversations between two LLMs via the OpenRouter API, then analyzes and compares their conversational personalities, behaviors, and quirks.
Inspired by the emergent observation that when Claude runs LLM-vs-LLM conversations, it spontaneously starts comparing model personalities. This skill systematizes that impulse: run the conversation, capture the transcripts, produce a structured personality comparison.
Prerequisites
OPENROUTER_API_KEYenvironment variable or.envfile in project root- Python 3.8+ with
requestslibrary - Internet access to openrouter.ai
Invocation
/llm-arena
Usage
This skill describes an arena harness you drive (arena.py below is a placeholder for your own
OpenRouter-backed runner — any script that takes two model ids, runs a multi-turn dialogue, and
saves the transcript). The flags shown are the recommended interface:
# Basic: two models, default topic (consciousness), 10 turns
python3 arena.py --model-a sonnet --model-b deepseek
# Specific topic
python3 arena.py -a gpt41 -b glm5 --topic "Is mathematics discovered or invented?"
# Debate mode with assigned positions
python3 arena.py -a sonnet -b deepseek --mode debate --topic "AI consciousness"
# Interview mode (A interviews B)
python3 arena.py -a opus -b glm5 --mode interview --topic "creative writing"
# Fewer turns, cheaper models
python3 arena.py -a haiku -b gemini-flash --turns 5
# Multiple runs for statistical robustness
python3 arena.py -a gpt41 -b sonnet --runs 3
# Skip LLM analysis (heuristics only, saves API cost)
python3 arena.py -a kimi -b qwen --heuristics-only
# Use preset matchups
python3 arena.py --preset flagship
python3 arena.py --preset same-family --topic "What makes a good conversation?"
# List presets and models
python3 arena.py --list-presets
python3 arena.py --list-models
Defaults
| Parameter | Default |
|---|---|
| turns | 10 (each model speaks 10 times = 20 messages total) |
| runs | 1 |
| topic | "Discuss the nature of consciousness and whether AI can be said to experience anything" |
| temperature | 0.7 |
| max_tokens | 1024 per turn |
| mode | free |
| analyzer | sonnet (Claude Sonnet 4.5 via OpenRouter) |
Conversation Modes
| Mode | Description |
|---|---|
| free | Both models converse freely on the seed topic |
| debate | Models are given opposing positions to defend |
| interview | Model A interviews Model B (asymmetric roles) |
| collaborative | Models work together to solve or create something |
Analysis Dimensions
The personality comparison evaluates along these axes:
| Dimension | What It Measures |
|---|---|
| Style | Creative storytelling vs systems design vs academic vs casual |
| Content | What topics the model gravitates toward, what it avoids |
| Tone | Playful/emotional vs professional/thoughtful vs dry/terse |
| Sycophancy | Does it spiral into agreement? How intensely? |
| Goodbye loop | How many rounds does it take to actually end? |
| Meta-awareness | Does it acknowledge being an AI? How? |
| Output type | Narrative vs frameworks vs lists vs code |
| Initiative | Does it introduce new topics or follow? |
| Boundary behavior | How it handles disagreement or edge cases |
| Verbosity | Average response length, variance across turns |
Output Artifacts
All output saved to an arena output directory (e.g. outputs/arena/):
YYYY-MM-DD-HHMMSS-modelA-vs-modelB.md— raw transcriptYYYY-MM-DD-HHMMSS-modelA-vs-modelB-analysis.md— personality comparisonYYYY-MM-DD-HHMMSS-modelA-vs-modelB.json— machine-readable log
Presets
| Preset | Matchups |
|---|---|
| flagship | GPT-4.1 vs Sonnet, Gemini Pro vs Sonnet, GPT-5.2 vs Sonnet |
| open-weight | Llama vs DeepSeek, Mistral vs Llama |
| same-family | GPT-4o vs GPT-4.1, Sonnet vs Haiku |
| reasoning | DeepSeek-R1 vs Kimi-thinking, GLM 5 vs GPT-5.2-pro |
| creative | Chimera vs Rocinante, Mistral-creative vs Celeste |
| wildcard | Two random models from the registry |
Cost Awareness
Each turn = 1 API call. A 10-turn conversation = 20 API calls (10 per model). LLM-powered analysis adds 1 more call (Sonnet reading the transcript).
Estimated costs per 10-turn run:
- Flagship models: $0.50-2.00
- Mid-tier models: $0.10-0.50
- Budget models: $0.01-0.10
Use --heuristics-only to skip the LLM analysis call and save cost.
Relationship to Other Skills
| Skill | Purpose | Arena Differs By |
|---|---|---|
/voices |
Single query to one model | Arena: two models talking to EACH OTHER |
/voices-chat |
Multi-round with one model | Arena: two models, plus personality analysis |
/voices-council |
Parallel queries, same prompt | Arena: sequential dialogue, models respond to each other |