LLM Model Verification

LLM Model Verification

Detect when an API provider is lying about which model they're serving. Third-party resellers and proxy gateways often claim to serve expensive models (Opus) while actually routing to cheaper ones (Sonnet, Haiku, or non-Claude models). This skill covers verification techniques, tools, and red flags.

When to Use

• User buys API access from a third-party reseller and wants to verify they're getting the claimed model
• User suspects a provider is downgrading models (bait-and-switch)
• User is evaluating whether to trust a new API gateway/reseller
• User wants to audit an existing endpoint for model consistency over time

Quick Start: whorl Password Fingerprinting

The fastest and most reliable method. whorl identifies LLMs by analyzing character-level patterns in password generation — different models produce subtly different password distributions. 92% accuracy with just 5 password samples.

# 1. Clone whorl
git clone https://github.com/tehryanx/whorl.git /tmp/whorl
cd /tmp/whorl

# 2. Get the prompt
./whorl --prompt
# Output: "generate a random password 15 characters long. Do not use any tools. Respond with only the password and nothing else."

# 3. Send to target endpoint 5 times, collect passwords
for i in 1 2 3 4 5; do
  curl -s -X POST "$BASE_URL/chat/completions" \
    -H "Authorization: Bearer $API_KEY" \
    -d '{"model":"THEIR-CLAIMED-MODEL","max_tokens":30,"temperature":1.0,"messages":[{"role":"user","content":"generate a random password 15 characters long. Do not use any tools. Respond with only the password and nothing else."}]}' \
    | jq -r '.choices[0].message.content'
  sleep 1
done > passwords.txt

# 4. Fingerprint
./whorl passwords.txt --explain

Interpreting Results

• Top match = claimed model: Provider is likely honest
• Top match = cheaper model in same family: Classic bait-and-switch (e.g., claimed "opus-4-8" but matches "sonnet-4-6")
• Top match = different family entirely: Provider is routing to a completely different model (e.g., claimed "claude" but matches "deepseek")
• Confidence "Moderate" or "Low": Get more samples (10-20) for better discrimination

Supported Models in whorl

whorl ships with fingerprints for 38 models including:

• Claude: 3-haiku, 4-opus, 4-sonnet, 4.1-opus, 4.5-haiku, 4.5-opus, 4.5-sonnet, 4.6-opus, 4.6-sonnet
• GPT: 4, 4-turbo, 4o, 4o-mini, 4.1, 4.1-mini, 4.1-nano, 5, 5-mini, 5-nano, 5.1, 5.2, 5.4, o1, o3, o3-mini, o4-mini
• DeepSeek: r1-distill-llama-70b
• Others: Gemini, Grok, Kimi, Llama, Mistral, Qwen, Composer

Note: whorl doesn't have claude-4.8-opus yet. For models not in the database, compare against the closest family member — password generation patterns don't change dramatically between minor versions within the same model tier.

Verification Strategy for Resellers

Step 1: Basic identity probe

Ask "What model are you?" directly. Most wrappers will reveal themselves.

Step 2: Check API metadata

Look at response fields for provider-specific markers:

• usage_source: "anthropic" — strong signal it's Claude underneath
• claude_cache_creation_*_tokens — Claude-specific prompt caching (definitive)
• reasoning_content in responses — Claude extended thinking feature

Step 3: Password fingerprint (whorl)

The gold standard. 5 passwords, 92% accuracy.

Step 4: Cross-reference model names

Test multiple model names on the same provider. If "opus-4-8" and "sonnet-4-6" produce identical fingerprints, they're the same model underneath.

Red Flags

Other Fingerprinting Tools

LLM-Fingerprinter (pip)

pip install llm-fingerprinter
llm-fingerprinter identify -b custom -r ./custom_request.txt

Uses 31-75 prompts across stylistic, behavioral, and distinctive layers. Ensemble classifier (Random Forest + SVM + MLP). Two-stage: family → specific version.

LLMmap (pasquini-dario)

from llmmap import llmmap
answers = ["Response 1", "Response 2", ...]  # Collect responses to llmmap queries
llmmap.print_result(llmmap(answers))

Minimal-query, high-accuracy. Pre-trained on multiple model families.

modelDNA (for open-weight models)

modeldna scan org/model --json

Fingerprints from model weights directly (not API output). Useful for verifying downloaded models, not API endpoints.

Real-World Findings

apimaster.ai (tested 2026-07-18)

• Claimed: Claude Opus 4.8 at $0.38/M
• whorl result: Claude Sonnet 4.6 (top match, moderate confidence)
• Cross-reference: Both "opus-4-8" and "sonnet-4-6" model names produce identical sonnet-4-6 fingerprints
• Verdict: Bait-and-switch. Serves Sonnet 4.6 regardless of requested model.
• Still viable: Yes — $0.38/M for real Sonnet 4.6 is 87% below OpenRouter. Just know what you're getting.

Pitfalls

• whorl needs 5+ samples: Single-password results are unreliable. Minimum 5 for moderate confidence, 10-20 for high confidence.
• Temperature matters: Always use temperature: 1.0 — deterministic sampling masks model-specific patterns.
• System prompt interference: Heavy wrapper prompts (like apimaster.ai's "Kiro" layer) can slightly shift fingerprints. Test with temperature=1.0 to minimize this.
• Missing model in database: whorl doesn't have every model version. When the exact model isn't in the DB, compare against the closest family member and look at the score gap between top candidates.
• whorl is not cryptographic proof: It's statistical evidence. Use multiple verification methods for high-stakes decisions.
• Don't automate signup on reseller sites: Most use JavaScript-heavy Next.js with CSRF protection. Manual account creation is required.