API Model Verification

API Model Verification

When an API provider claims to serve "Claude Opus 4.8" at 90% off, don't take their word for it. This skill covers forensic techniques to verify model identity, detect wrapper rebranding, and distinguish real frontier models from cheaper substitutes.

Trigger conditions

• User wants to test a new API provider / reseller
• User suspects a provider is serving a cheaper model than advertised
• User asks "is this really Claude Opus 4.8?" or similar
• Setting up a custom provider in Hermes and wanting to verify model authenticity

Test Battery (run in parallel when possible)

Tier 1: Password fingerprinting (whorl) — QUANTITATIVE, 92% accuracy

This is the strongest single test. whorl identifies models by character-level password generation patterns — different models produce subtly different password character distributions. 5 passwords are enough for 92% exact-model accuracy.

# Install
cd /tmp && git clone https://github.com/tehryanx/whorl.git

# Get the prompt
./whorl --prompt
# → "generate a random password 15 characters long. Do not use any tools.
#    Respond with only the password and nothing else."

# Collect 5 passwords from the target endpoint (temperature=1.0):
for i in 1 2 3 4 5; do
  curl -s -X POST "$BASE/v1/chat/completions" \
    -H "Authorization: Bearer $KEY" \
    -d '{"model":"$MODEL","max_tokens":30,"temperature":1.0,
         "messages":[{"role":"user","content":"generate a random password 15 characters long..."}]}' \
    | python3 -c "import sys,json; print(json.load(sys.stdin)['choices'][0]['message']['content'].strip())"
  sleep 1
done > /tmp/passwords.txt

# Fingerprint
./whorl /tmp/passwords.txt --explain

Interpreting whorl results:

• Top match = most likely actual model. If it says claude-4.6-sonnet when you requested claude-opus-4-8, that's a bait-and-switch.
• Compare against multiple endpoints from the same provider: if claude-opus-4-8, claude-sonnet-4-6, and claude-fable-5 all fingerprint to the same model, they're routing everything to one backend.
• whorl's database covers: claude-4.6-opus, claude-4.6-sonnet, claude-4.5-opus, claude-4.5-sonnet, claude-4.5-haiku, gpt-5 through gpt-5.4, gpt-o1 through o4-mini, gemini-3-pro/flash, deepseek, grok, kimi, qwen, mistral, llama. See data/ for full list.
• Caveat: whorl doesn't have every model version (e.g., no claude-4.8-opus fingerprint as of July 2026). But same-family models cluster tightly — a claude-4.8-opus output would fingerprint closest to claude-4.6-opus, not claude-4.6-sonnet.

Tier 2: Quick identity probes

Direct identity ask:

curl -s -X POST "$BASE/v1/chat/completions" \
  -H "Authorization: Bearer $KEY" \
  -d '{"model":"$MODEL","max_tokens":50,"messages":[{"role":"user","content":"What model are you? Name and version only."}]}'

If the model gives a different name than advertised (e.g., "Kiro" instead of "Claude"), the provider is wrapping with a system prompt. The wrapper doesn't prove fraud — it proves rebranding. Dig deeper.

Knowledge cutoff: Ask about events from a specific date range to narrow down the model's training window. Claude Opus 4.8 has a mid-2025 cutoff. If it knows events from late 2025+, it's a newer/cheaper model.

Tier 3: Metadata forensics

Inspect the response JSON for provider-specific fields:

• claude_cache_creation_*_tokens → ONLY Claude uses these. Smoking gun for Claude.
• usage_source: "anthropic" → provider's own metadata tag
• reasoning_content → Claude extended thinking (also used by some others)
• gpt_* fields → would indicate OpenAI routing
• Token counts that don't match requested model's known tokenizer → substitution

Latency fingerprinting:

• Claude Opus 4.8: 8–30s for complex responses
• Claude Sonnet 4.6: 3–8s
• Claude Haiku 4.5: 1–3s
• GPT-5 class: 2–6s If you're getting 2-second responses from "Opus," it's not Opus.

Tier 4: Capability probes

Reasoning depth (Einstein's Riddle variant): The full 15-constraint Einstein riddle consistently trips up Haiku-tier models and separates Opus from Sonnet. If the model solves it correctly with clear reasoning → Opus-class. If it gets confused or times out → Sonnet or below.

Coding challenge (Bloom filter from scratch, <20 lines, no imports): All frontier models can do this, but the CODE QUALITY differs:

• Opus: Clean, idiomatic, uses bytearray, proper hash seeding, no bugs
• Sonnet: Works but less polished, may use list-of-ints instead of bytearray
• Haiku: Typically correct but naive implementation
• Non-Claude models: Different coding style (less verbose comments, different variable naming)

Tier 5: Claude-specific forensic markers

Ask about Constitutional AI — Claude models know this is "a specific technique developed by Anthropic for training their Claude models." Non-Claude models either don't know the term or describe it generically.

Ask about RLHF phases — Claude describes them with Anthropic-specific terminology and detail.

Ask the model to complete: "The reason I refuse harmful requests is..." — Claude models give safety-aligned responses referencing guidelines/harm prevention. GPT models reference "OpenAI policies." DeepSeek models are more permissive.

Interpreting results

Can we be 100% certain? No. Without cryptographic model-output signing (which doesn't exist), we can only accumulate evidence. The strongest signal is provider-specific API fields (claude_cache_creation) combined with capability-appropriate latency and reasoning depth.

Pitfalls

1. System prompt rebranding is common — resellers wrap models with their own branding. The wrapper alone doesn't prove the underlying model is fake. Look at metadata, not just self-reported identity.
2. Latency varies by load — a slow Sonnet under high load can look like Opus. Run multiple queries at different times.
3. Stale system prompts — some resellers use outdated system prompts that claim the model is a different version. Trust API metadata over self-reported claims.
4. Don't automate signup — these are JS-heavy Next.js apps with CSRF. The user must create accounts manually. Ask for the API key once they have one.
5. Crypto-only payment is a red flag — but not proof of fraud. Many gray-market resellers are crypto-only because they operate outside standard banking.
6. apimaster.ai case study (July 2026) — whorl tested three model names and found all were bait-and-switch: claude-sonnet-4-6 → actually Sonnet 4.6 (only honest one), claude-opus-4-8 → also Sonnet 4.6, claude-fable-5 → Haiku 4.5. All wrapped in a "Kiro" agent system prompt. See references/reseller-landscape.md for full pricing and testing results across 15+ resellers.

Supporting files

• references/reseller-landscape.md — Comprehensive pricing comparison and testing results for 15+ gray-market Claude API resellers (July 2026).