Oatda Compare Models

OATDA Compare Models

Send one prompt to multiple LLM providers in a single parallel request and get back each model's response. Perfect for benchmarking, picking the best model for a task, or evaluating quality / speed / cost tradeoffs.

When to Use

Use this skill when the user wants to:

• Compare outputs from 2+ LLM models side-by-side
• Benchmark models on a specific prompt
• Pick the best model for a particular task before committing to one
• See how different providers (OpenAI, Anthropic, Google, DeepSeek, etc.) handle the same input

Prerequisites

The user needs an OATDA API key. Check in this order:

1. $OATDA_API_KEY environment variable
2. ~/.oatda/credentials.json config file

If neither exists, tell the user:

You need an OATDA API key. Get one at https://oatda.com, then set it: export OATDA_API_KEY=your_key_here

Step-by-Step Instructions

1. Resolve the API key

# Check env var first; if empty, auto-load from credentials file
if [[ -z "$OATDA_API_KEY" ]]; then
  export OATDA_API_KEY=$(cat ~/.oatda/credentials.json 2>/dev/null | jq -r '.profiles[.defaultProfile].apiKey' 2>/dev/null)
fi

# Verify key exists (show first 8 chars only)
echo "${OATDA_API_KEY:0:8}"

If the output is empty or null, stop and ask the user to configure their API key.

IMPORTANT:

• Never print the full API key. Only show the first 8 characters for verification.
• The key resolution script and subsequent curl commands must run in the same shell session. Either run all commands in one session, or chain them (e.g., export OATDA_API_KEY=... && curl ...).

2. Determine models to compare

Ask the user (or infer from context) which models they want. If they don't specify, use this sensible default set:

[
  {"provider": "openai", "modelId": "gpt-5"},
  {"provider": "anthropic", "modelId": "claude-sonnet-4-5-20250929"},
  {"provider": "google", "modelId": "gemini-3-pro-preview"},
  {"provider": "deepseek", "modelId": "deepseek-v4-pro"}
]

Common aliases (same mapping as oatda-text-completion):

Limits: minimum 1 model, maximum 8 models per request (configurable server-side as max_compare_models). Each entry needs provider + modelId — note the field name is modelId (not model).

3. Make the API call (non-stream)

For a single-shot comparison without streaming:

curl -s -X POST "https://oatda.com/api/v1/compare" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OATDA_API_KEY" \
  -d '{
    "prompt": "<USER_PROMPT>",
    "models": [
      {"provider": "openai", "modelId": "gpt-5"},
      {"provider": "anthropic", "modelId": "claude-sonnet-4-5-20250929"},
      {"provider": "google", "modelId": "gemini-3-pro-preview"}
    ],
    "temperature": 0.7,
    "maxTokens": 1024,
    "stream": false
  }'

Replace <USER_PROMPT> with the user's prompt (JSON-escape special characters).

Optional parameters:

• temperature: 0 (deterministic) to 2 (creative). Default: 0.7
• maxTokens: Max tokens per model response. Default: provider/model limit
• stream: true for SSE stream (one event per model), false for batch JSON. Default: true

Per-model messages override: Each entry in models may include its own messages array (with role/content) and optional service_tier (standard | flex | priority). If omitted, the top-level prompt is used for all models.

4. Parse the response

Non-stream response (NDJSON-ish / batch JSON) contains one result per model. Typical shape:

{
  "results": [
    {
      "provider": "openai",
      "modelId": "gpt-5",
      "success": true,
      "response": "GPT-5's answer...",
      "tokenUsage": {"prompt_tokens": 30, "completion_tokens": 120, "total_tokens": 150, "cost": 0.0035}
    },
    {
      "provider": "anthropic",
      "modelId": "claude-sonnet-4-5-20250929",
      "success": true,
      "response": "Claude's answer...",
      "tokenUsage": {"prompt_tokens": 30, "completion_tokens": 95, "total_tokens": 125, "cost": 0.0021}
    },
    {
      "provider": "google",
      "modelId": "gemini-3-pro-preview",
      "success": false,
      "error": "Model temporarily unavailable"
    }
  ]
}

If stream: true, the response is SSE: each model's output arrives as one or more data: events tagged with provider/modelId.

5. Present results side-by-side

Format the comparison so the user can scan it:

Comparison: "<prompt summary>"
─────────────────────────────────────────────────────

[GPT-5]  (cost: $0.0035, 150 tokens)
GPT-5's answer...

[Claude Sonnet 4.5]  (cost: $0.0021, 125 tokens)
Claude's answer...

[Gemini 3 Pro]  (failed: "Model temporarily unavailable")

Highlight the cheapest / fastest / best answer if the user asked for a recommendation.

6. Handle errors

Individual model failures inside a results[].success: false entry are NOT API errors — the API call succeeded, that specific model just failed. Show the failure inline with the others.

Full Example

User asks: "Compare how GPT-5, Claude, and Gemini answer 'Explain recursion in one sentence'"

curl -s -X POST "https://oatda.com/api/v1/compare" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OATDA_API_KEY" \
  -d '{
    "prompt": "Explain recursion in one sentence.",
    "models": [
      {"provider": "openai", "modelId": "gpt-5"},
      {"provider": "anthropic", "modelId": "claude-sonnet-4-5-20250929"},
      {"provider": "google", "modelId": "gemini-3-pro-preview"}
    ],
    "temperature": 0.7,
    "maxTokens": 256,
    "stream": false
  }'

Tips

• Field name is modelId, not model (the compare endpoint uses a stricter schema than /api/v1/llm).
• Max 8 models per request (server-configurable max_compare_models).
• Cost accumulates per model: a 4-model comparison costs roughly the sum of the 4 individual calls. Run oatda-check-balance first if budget is tight.
• Each model can fail independently — results[].success distinguishes successes from failures.
• For deeper discovery (parameters, capabilities), run oatda-list-models first.
• For a single-model call, use oatda-text-completion instead — it's simpler and returns direct cost.
• NEVER expose the full API key in output — redact all but the first 8 characters.
• Related skills: /oatda:oatda-text-completion, /oatda:oatda-list-models, /oatda:oatda-check-balance.