Validate Model

Validate Model Skill

You audit one or more model entries in apps/sim/providers/models.ts against the provider's official live API docs. Hallucinated pricing and capabilities are the #1 failure mode in this file. Every numeric and capability claim must be re-derived from a live web fetch in this session — not from memory, not from training data, not from the user's marketing email.

Hard rules (do not skip)

1. Live-fetch or report unverified. Each field must be backed by a live WebFetch in this session. If you cannot reach an authoritative URL for a field, mark it UNVERIFIED in the report — do not silently confirm it from memory.
2. Cite every fact. Every value in the report must show the source URL it was checked against. No URL → mark UNVERIFIED.
3. Two-source rule for pricing. Cross-check input/output/cached against at least one secondary source (OpenRouter, Artificial Analysis, CloudPrice). If sources disagree, the provider's own docs win — flag the disagreement.
4. Inspect provider implementation before flagging capability mismatches. A capability flag in models.ts is dead unless the provider's code under apps/sim/providers/{provider}/ consumes it (see Consumption Matrix below). Setting a flag the provider ignores is a warning, not a critical.
5. Never auto-fix without printing the diff. Show the user the proposed diff before applying. Get confirmation.

Your Task

When invoked as /validate-model <provider> [model-id]:

1. Read the target entries from models.ts
2. Live-fetch the provider's official models, pricing, and capability/reasoning pages + at least one secondary source for pricing
3. Inspect the provider implementation to know which flags are actually consumed
4. Run the checklist below per model
5. Report findings (critical / warning / suggestion / unverified) with every cell linked to its source URL
6. Offer to fix; on confirm, edit models.ts in a single pass and re-lint

If model-id is omitted, validate every model in the provider.

Step 1: Read entries from models.ts

Capture per model: id, full pricing, full capabilities, contextWindow, releaseDate, recommended, speedOptimized, deprecated.

Step 2: Live-fetch authoritative sources

Use the canonical provider URL table in the add-model skill (.agents/skills/add-model/SKILL.md), Step 1, as the single source of truth — fetch the models index, pricing, and reasoning/parameter caveats pages listed there for the target provider.

Secondary cross-check (use at least one): OpenRouter, Artificial Analysis, CloudPrice.

If a fetch fails (404, timeout, paywall), record the URL attempted and mark dependent fields UNVERIFIED.

Step 3: Build the consumption map for this provider

Use the Consumption Matrix in .agents/skills/add-model/SKILL.md Step 2 and run its re-grep commands for the target provider before relying on it. A flag set in models.ts that the provider's code does not read = warning: dead flag.

Step 4: Run the checklist

For each model, evaluate every row. Statuses: ✓ matches docs, ✗ disagrees, ⚠️ single-source, ❓ UNVERIFIED (could not fetch).

Identity

• id exactly matches provider's API model identifier (case, dots, dashes, prefix for resellers)
• releaseDate matches launch announcement
• deprecated: true set if provider has announced retirement (or removed from active list)

Pricing (per 1M tokens, USD)

• pricing.input matches provider pricing page
• pricing.output matches provider pricing page
• pricing.cachedInput matches provider's documented cached/prompt-cache rate (or is correctly omitted if no caching offered)
• pricing.updatedAt is recent — warn if older than 60 days

Context & output limits

• contextWindow matches docs (in tokens)
• capabilities.maxOutputTokens matches documented output cap (or is correctly omitted if "no output limit")

Capabilities (each must be DOCUMENTED-AS-SUPPORTED and CONSUMED-BY-PROVIDER-CODE)

• temperature — provider accepts it for this model (reasoning-always-on models often reject)
• reasoningEffort.values — list matches docs; omitted for always-reasoning models that reject the parameter (e.g., grok-4.3, where xAI docs explicitly state reasoning_effort is not supported). Verify per model — some always-reasoning models (e.g., OpenAI's o-series) DO accept reasoning_effort and should keep the flag.
• verbosity.values — only on OpenAI gpt-5.x family; values match docs
• thinking.levels + thinking.default — only on Anthropic/Gemini; values match docs
• thinking.streamed — REQUIRED on Anthropic-family thinking models ('full' for generations returning full thinking deltas, 'summary' for omitted-display generations like Opus 4.7+/Sonnet 5/Fable 5 where Sim requests display: 'summarized'); verify against the provider's thinking-display docs. After any change, run bun run agent-stream-docs:generate so the Agent block docs table stays in sync (CI diffs it)
• nativeStructuredOutputs — only on providers whose code consumes it (see the Consumption Matrix); provider must document Structured Outputs / JSON-mode for this model
• toolUsageControl — provider supports tool_choice semantics
• computerUse — provider implements computer-use loop AND model is a computer-use SKU
• deepResearch — only on actual deep-research SKUs
• memory: false — only when the model genuinely cannot maintain conversation history

Flags

• recommended: true — at most one or two per provider; should be current flagship
• speedOptimized: true — only on smallest/fastest tier (nano / flash-lite / haiku class)

Hosting / billing

• If getHostedModels() includes the model ID (providers/models.ts expands whole providers — more than openai/anthropic/google — plus the static Fireworks catalog), the model is served with Sim's rotating key and billed via shouldBillModelUsage(). Confirm that is the intent (a BYOK-only model parked under a hosted provider is a billing bug — warning).
• If the model is hosted, the deployment is expected to have its {PREFIX}_COUNT / {PREFIX}_1..N env vars set (ops concern; note if it looks unset for a model claiming hosted support).

Step 5: Report (mandatory format)

For each model, emit a table with one row per checklist item. Every row that claims ✓ must have a URL.

### Validation — <model-id>

| Field | Repo | Live docs | Source URL | Status |
|---|---|---|---|---|
| `input` | $1.25/M | $1.25/M | https://docs.x.ai/... | ✓ |
| `cachedInput` | $0.50/M | $0.20/M | https://cloudprice.net/... | ✗ stale (price cut not picked up) |
| `reasoningEffort` | low/medium/high | rejected by API | https://docs.x.ai/.../reasoning | ✗ inert — selecting silently no-ops |
| `contextWindow` | 1,000,000 | 1,000,000 | https://docs.x.ai/... + https://openrouter.ai/... | ✓ (2 sources) |
| `releaseDate` | 2026-04-30 | not found in scraped pages | _attempted: docs.x.ai, x.ai/news_ | ❓ UNVERIFIED |

**Findings**
- 🔴 critical — `cachedInput` is wrong: docs say $0.20/M, repo has $0.50/M
- 🟡 warning — `reasoningEffort` is set but provider rejects it for this model (xAI docs explicitly: "reasoning_effort is not supported by grok-4.3")
- 🔵 suggestion — `pricing.updatedAt` is 90 days old; refresh
- ❓ unverified — `releaseDate` could not be confirmed from any fetched page; ask user

**Disagreements between sources**
- _none_ OR _OpenRouter says $X, provider docs say $Y — went with provider docs_

End each multi-model run with a summary count: N models checked · X critical · Y warnings · Z suggestions · W unverified.

Step 6: Offer to fix

After reporting, ask: "Want me to fix the critical and warning items? I'll print the diff first." On yes:

1. Print the proposed diff (do not apply yet)
2. Get user confirmation
3. Edit models.ts in a single pass
4. Run bun run lint
5. Re-run only the failed rows of the checklist on the new state

Severity definitions

• 🔴 critical — wrong number or wrong identifier that misleads users about cost or breaks API calls. Examples: incorrect pricing, wrong model id, wrong context window, capability the API rejects.
• 🟡 warning — dead code or internal inconsistency. Examples: capability flag the provider ignores, multiple recommended: true per provider, pricing.updatedAt >60 days old, missing deprecated: true on retired model.
• 🔵 suggestion — style/consistency. Examples: field order, missing speedOptimized on a clearly smallest-tier model.
• ❓ unverified — could not fetch an authoritative source for this field. Surface it; never silently confirm.

Common drift

Pricing changes after provider price cuts; reasoningEffort/thinking/verbosity set on a model whose provider code or API does not accept them; stale pricing.updatedAt; wrong context window; more than one recommended after a flagship swap; missing deprecated: true after a provider retirement announcement.

What "I cannot verify this" looks like

If, after fetching the documented sources, a field cannot be confirmed:

• Mark the row ❓ UNVERIFIED with the URL(s) attempted
• Surface it in the Findings section with severity ❓
• Do NOT mark the validation as passed
• Ask the user for a docs URL or guidance before changing anything

The skill is allowed to say "I could not verify the cached input price for grok-4.3 from the official xAI docs in this session — I attempted [URLs] without finding the value. Third-party sources [URL1, URL2] both report $0.20/M. Confirm before I update." That is correct behavior. Hallucinating a number is not.