Benchmark Models

When to invoke

Use when: "benchmark models", "compare models", "which model is best for X", "cross-model comparison", "model shootout".

Preamble

eval "$(~/.vibestack/bin/vibe-slug 2>/dev/null)" 2>/dev/null || SLUG="unknown"
_LEARN_FILE="${VIBESTACK_HOME:-$HOME/.vibestack}/projects/${SLUG:-unknown}/learnings.jsonl"
if [ -f "$_LEARN_FILE" ]; then
  _LEARN_COUNT=$(wc -l < "$_LEARN_FILE" 2>/dev/null | tr -d ' ')
  echo "LEARNINGS: $_LEARN_COUNT entries loaded"
  if [ "$_LEARN_COUNT" -gt 5 ] 2>/dev/null; then
    ~/.vibestack/bin/vibe-learnings-search --limit 5 2>/dev/null || true
  fi
else
  echo "LEARNINGS: none yet"
fi

{{include lib/snippets/session-host.md}}

{{include lib/snippets/decision-brief.md}}

{{include lib/snippets/working-protocols.md}}

{{include lib/snippets/state-protocols.md}}

/benchmark-models — Cross-Model Skill Benchmark

Different from /benchmark — that skill measures web page performance (Core Web Vitals, load times). This skill measures AI model performance on skills or arbitrary prompts.

Step 0: Locate the binary

BIN="$HOME/.vibestack/bin/vibe-model-benchmark"
[ -x "$BIN" ] || { echo "ERROR: model benchmark binary not found." >&2; exit 1; }
echo "BIN: $BIN"

If not found, stop and tell the user: "vibe-model-benchmark is required for this skill but is not installed at ~/.vibestack/bin/vibe-model-benchmark. vibestack does not bundle this binary — it's a separate dependency. See docs/external-tools.md for current options."

Step 1: Choose a prompt

Use AskUserQuestion with the preamble format:

• Re-ground: current project + branch.
• Simplify: "A cross-model benchmark runs the same prompt through 2-3 AI models and shows you how they compare on speed, cost, and output quality. What prompt should we use?"
• RECOMMENDATION: A because benchmarking against a real skill exposes tool-use differences, not just raw generation.
• Options:
  • A) Benchmark one of my skills (we'll pick which skill next). Completeness: 10/10.
  • B) Use an inline prompt — type it on the next turn. Completeness: 8/10.
  • C) Point at a prompt file on disk — specify path on the next turn. Completeness: 8/10.

If A: list skills that have SKILL.md files (from find ~/.claude/skills -name SKILL.md -not -path '*/vibestack/*'), ask the user to pick one via a second AskUserQuestion. Use the picked SKILL.md path as the prompt file.

If B: ask the user for the inline prompt. Use it verbatim via --prompt "<text>".

If C: ask for the path. Verify it exists. Use as positional argument.

Step 2: Choose providers

"$BIN" --prompt "unused, dry-run" --models claude,gpt,gemini --dry-run

Show the dry-run output. The "Adapter availability" section tells the user which providers will actually run (OK) vs skip (NOT READY — remediation hint included).

If ALL three show NOT READY: stop with a clear message — benchmark can't run without at least one authed provider. Suggest claude login, codex login, or gemini login / export GOOGLE_API_KEY.

If at least one is OK: AskUserQuestion:

• Simplify: "Which models should we include? The dry-run above showed which are authed. Unauthed ones will be skipped cleanly — they won't abort the batch."
• RECOMMENDATION: A (all authed providers) because running as many as possible gives the richest comparison.
• Options:
  • A) All authed providers. Completeness: 10/10.
  • B) Only Claude. Completeness: 6/10 (no cross-model signal — use /ship's review for solo claude benchmarks instead).
  • C) Pick two — specify on next turn. Completeness: 8/10.

Step 3: Decide on judge

[ -n "$ANTHROPIC_API_KEY" ] || grep -q 'ANTHROPIC' "$HOME/.claude/.credentials.json" 2>/dev/null && echo "JUDGE_AVAILABLE" || echo "JUDGE_UNAVAILABLE"

If judge is available, AskUserQuestion:

• Simplify: "The quality judge scores each model's output on a 0-10 scale using Anthropic's Claude as a tiebreaker. Adds ~$0.05/run. Recommended if you care about output quality, not just latency and cost."
• RECOMMENDATION: A — the whole point is comparing quality, not just speed.
• Options:
  • A) Enable judge (adds ~$0.05). Completeness: 10/10.
  • B) Skip judge — speed/cost/tokens only. Completeness: 7/10.

If judge is NOT available, skip this question and omit the --judge flag.

Step 4: Run the benchmark

Construct the command from Step 1, 2, 3 decisions:

"$BIN" <prompt-spec> --models <picked-models> [--judge] --output table

Where <prompt-spec> is either --prompt "<text>" (Step 1B), a file path (Step 1A or 1C), and <picked-models> is the comma-separated list from Step 2.

Stream the output as it arrives. This is slow — each provider runs the prompt fully. Expect 30s-5min depending on prompt complexity and whether --judge is on.

Step 5: Interpret results

After the table prints, summarize for the user:

• Fastest — provider with lowest latency.
• Cheapest — provider with lowest cost.
• Highest quality (if --judge ran) — provider with highest score.
• Best overall — use judgment. If judge ran: quality-weighted. Otherwise: note the tradeoff the user needs to make.

If any provider hit an error (auth/timeout/rate_limit), call it out with the remediation path.

Step 6: Offer to save results

AskUserQuestion:

• Simplify: "Save this benchmark as JSON so you can compare future runs against it?"
• RECOMMENDATION: A — skill performance drifts as providers update their models; a saved baseline catches quality regressions.
• Options:
  • A) Save to ~/.vibestack/benchmarks/<date>-<skill-or-prompt-slug>.json. Completeness: 10/10.
  • B) Just print, don't save. Completeness: 5/10 (loses trend data).

If A: re-run with --output json and tee to the dated file. Print the path so the user can diff future runs against it.

Important Rules

• Never run a real benchmark without Step 2's dry-run first. Users need to see auth status before spending API calls.
• Never hardcode model names. Always pass providers from user's Step 2 choice — the binary handles the rest.
• Never auto-include --judge. It adds real cost; user must opt in.
• If zero providers are authed, STOP. Don't attempt the benchmark — it produces no useful output.
• Cost is visible. Every run shows per-provider cost in the table. Users should see it before the next run.

Capture Learnings

Before finishing, review the session for durable learnings and log each one. This review always runs — it is not conditional on something having felt noteworthy. Phrased as a condition it reads as optional and gets skipped, which is why almost every recorded learning arrives via an explicit /learn instead.

~/.vibestack/bin/vibe-learnings-log '{"skill":"benchmark-models","type":"TYPE","key":"SHORT_KEY","insight":"DESCRIPTION","confidence":N,"source":"SOURCE","files":["path/to/relevant/file"]}'

Types: pattern, pitfall, preference, architecture, operational.

Log only genuine discoveries — provider auth quirks, cost surprises, a prompt shape that skews one model's numbers. If the review turns up nothing durable, say so explicitly rather than staying silent, so a skipped step and an empty one are distinguishable.