Fill Model Descriptions

Fill Model Descriptions

Goal

Keep descriptions in packages/llm-info/data/models.yml concise and accurate. Fill empty descriptions and refresh existing descriptions when newly added models or authoritative documentation make them obsolete. Source descriptions from OpenRouter first, then provider documentation as a fallback.

Scope and Contract

• Fill every empty description for which a confident source is available.
• Preserve human-curated, non-empty descriptions unless a newly added model or authoritative source makes a concrete claim obsolete. Typical obsolete claims include "latest," "most capable," availability, or capabilities that have demonstrably changed.
• Do not rewrite existing descriptions only for style, tone, or consistency. When an existing description must change, make the smallest factual correction and record it in the final summary.
• The file is a top-level YAML map keyed by provider id (anthropic:, openai:, …). Each provider's value is an array of model entries.
• Do not modify any other field, reorder entries, or change formatting (flow-style arrays like roles: [chat, edit], blank lines between entries, blank line between provider sections).
• This skill requires network access.

Workflow

1. Read packages/llm-info/data/models.yml. Parse it with the yaml library's parseDocument (NOT parse) so formatting and comments are preserved on write-back.

2. Collect and review descriptions:

   • Build one worklist of { provider, model } pairs containing every entry where description is empty.
   • Review related existing entries in the same model families for claims made obsolete by the new entries. Also review descriptions affected by a factual change found in authoritative documentation, and add every obsolete entry to the same worklist.

3. Source descriptions in this priority order:

   a. OpenRouter — GET https://openrouter.ai/api/v1/models (no auth required). Build a lookup keyed by lowercased id. For each { provider, model }, try candidates in this order:

   • ${vendor}/${model} where vendor comes from the Vendor Map below.
   • ${vendor}/${model.replaceAll('-', '.')} (some OR ids use . separators, e.g. claude-3.5-sonnet).
   • For Bedrock entries with region/alias prefixes (us., eu., jp., global., anthropic., minimax., mistral., zai., nvidia.), strip the prefix and retry with the appropriate vendor.
   • For Google entries with @default suffix (e.g. claude-opus-4-7@default), strip the suffix and try anthropic/${rest}. These are Vertex-hosted Anthropic models — descriptions should note they are Vertex AI only (not Gemini API).

   b. Provider docs — if no OpenRouter hit, search the canonical provider's docs page (anthropic.com/news, platform.openai.com/docs/models, ai.google.dev/gemini-api/docs/models, mistral.ai/news, x.ai/blog, etc.) for the model's one-line summary. Use the WebSearch or WebFetch tool.

   c. Skip — if neither source has a confident answer, leave description: "" and record the { provider, model } for the final summary. Empty is better than wrong.

4. Truncate each sourced description per the rules in the "Truncation" section.

5. Write the YAML back using targeted Pair mutations on the parsed Document, not text replacement. Reference implementation: packages/llm-info/src/sync-models.ts shows how to load a Document, mutate sequence/map nodes, and stringify with { lineWidth: 0, flowCollectionPadding: false } to preserve formatting.

6. Validate by running:

   pnpm --filter @marimo-team/llm-info test
   

   The schema.test.ts test will catch any structural drift.

7. Report to the user:

   • How many empty descriptions were filled.
   • How many obsolete descriptions were updated, with the provider/model list and the factual reason.
   • How many were skipped (with the provider/model list).

Vendor Map

OpenRouter id vendor prefixes for each marimo provider:

Truncation

• One sentence. Split on . and take the first. Strip trailing whitespace and any markdown formatting (**, _, backticks).
• Hard cap at 200 characters. If the first sentence is longer, cut at the last word boundary before 200 and append ….
• Strip leading articles for consistency: prefer "Frontier reasoning model optimized for…" over "This is a frontier reasoning model optimized for…".
• Prefer capability statements over marketing. If OpenRouter gives "the best model from X" but the provider docs give "optimized for tool use and long-context reasoning", prefer the latter.

Examples

Good (matches the tone of existing hand-curated entries):

• Opus model optimized for coding and long-running professional tasks
• Sonnet-class model with strong performance across coding, agents, and professional work
• Lightweight Gemma model trained by Google, designed to run on a single GPU
• Multimodal Mixture-of-Experts model optimized for complex tool use and reasoning

Bad (do not produce these):

• Claude 3.5 Sonnet delivers better-than-Opus capabilities, faster-than-Sonnet speeds, at the same Sonnet prices. Sonnet is particularly good at coding, vision… — too long, contains marketing
• The best model from Anthropic — vague, no information content
• An AI model — useless
• **Claude 3.5 Sonnet** is a _frontier_ model… — markdown not stripped

Quick Reference

• Data file: packages/llm-info/data/models.yml
• Schema: packages/llm-info/src/index.ts (AiModel interface)
• YAML write reference: packages/llm-info/src/sync-models.ts (buildEntryNode, addProviderSection)
• OpenRouter API: https://openrouter.ai/api/v1/models (public, no auth)
• Validation: pnpm --filter @marimo-team/llm-info test