Karvey Benchmark Models

Karvey — Model Benchmark (cross-model)

CROSS-CUTTING SKILL of the Karvey Method. It is a support layer, NOT a phase. It does not advance or modify the method's cycle: it never touches spec.json:phase. It can be invoked at any time without altering the project's state.

Inspired by gstack /benchmark-models: a side-by-side comparison of models (for example Claude vs GPT vs Gemini) over the same skill or task, measuring latency, tokens, cost, and, optionally, quality judged by an LLM.

Purpose

Decide with data which model is best for a given task or skill in the project. Instead of choosing a model by intuition or habit, this skill runs the same task across several models and delivers an objective comparison table plus a well-founded recommendation.

It is a meta / diagnostic skill: it observes and measures, but it does not modify the project's code or produce artifacts from the method's phases. It generates no commits, does not change the solution's files, does not advance the phase.

Steps

1. Define the task and the models to compare.

   • Identify the target task or skill (from the argument or by asking the user).
   • Detect the model CLIs available in the environment: claude, codex/gpt, gemini, etc.
   • Degrade gracefully: if a CLI is not accessible, exclude it from the benchmark and warn. Compare only the models actually available.

2. Run the same task on each model.

   • Use exactly the same prompt/input for all, so the comparison is fair.
   • Run in isolation per model and capture the full output of each one.

3. Measure latency, tokens, and cost.

   • Latency: wall-clock time of each run.
   • Tokens: input + output reported by each CLI.
   • Cost: estimate from tokens and each model's current price.

4. (Optional) Quality judged by an LLM-judge.

   • If quality needs to be evaluated, use a model as judge to score the outputs against an agreed rubric (accuracy, completeness, format, etc.).
   • Keep the judge fixed and the rubric explicit so the scores are comparable.

5. Comparison table + recommendation.

   • Present a side-by-side table: model, latency, tokens, cost, and (if applicable) quality.
   • Close with a clear recommendation of which model is best for that task and why (cost/quality/latency balance per the objective).

Reminders

• Does not advance the phase. This skill never writes spec.json:phase or triggers method transitions.
• Does not touch the project's code. It only reads, runs test runs, and reports.
• It is invocable at any point in the Karvey cycle as support for decision-making.

Part of the Karvey™ Method — © HainTech, by Mauricio Quezada Ibáñez · Apache 2.0 · see karvey/LICENSE and karvey/TRADEMARK.md.