Model Manager

Model Manager

Test API access, validate configurations, and add new AI models to the AILANG eval suite.

Quick Start

Most common usage:

# User says: "Can we add GPT-5.1 to the eval suite?"
# This skill will:
# 1. Test API access to GPT-5.1
# 2. Find the correct API model name
# 3. Look up pricing information
# 4. Update models.yml configuration
# 5. Run a test benchmark to verify

When to Use This Skill

Invoke this skill when:

• User asks to "add a new model" to eval suite
• User mentions checking if a model is "accessible" or "available"
• User wants to "test API access" to a model
• User asks to "update models.yml" or "check pricing"
• User says "can we use [model name]?" for evaluations

Available Scripts

scripts/test_model_access.sh <provider> <model-name>

Test API access to a model and display authentication status.

Usage:

# Test OpenAI model
scripts/test_model_access.sh openai gpt-5.1

# Test Anthropic model
scripts/test_model_access.sh anthropic claude-sonnet-4-5-20250929

# Test Google Gemini via Vertex AI
scripts/test_model_access.sh google gemini-3-pro-preview-11-2025

Output:

Testing: openai/gpt-5.1
✓ OPENAI_API_KEY found
✓ API call successful
✓ Model: gpt-5.1-2025-11-13
✓ Tokens: 13 input, 10 output (10 reasoning)
Ready to add to models.yml

scripts/find_model_info.sh <model-keywords>

Search for model information using web search and return API names + pricing.

Usage:

# Find GPT-5.1 info
scripts/find_model_info.sh "GPT-5.1 API model name pricing"

# Find Gemini 3 Pro info
scripts/find_model_info.sh "Gemini 3 Pro API documentation"

Output:

Searching for: GPT-5.1 API model name pricing
✓ Found API names:
  - gpt-5.1 (Thinking mode)
  - gpt-5.1-chat-latest (Instant mode)
✓ Pricing:
  Input: $1.25 per 1M tokens
  Output: $10.00 per 1M tokens
  Cached: $0.125 per 1M tokens

scripts/update_models_yml.sh <friendly-name> <api-name> <provider> <input-price> <output-price>

Add a new model to models.yml configuration.

Usage:

# Add GPT-5.1
scripts/update_models_yml.sh \
  gpt5-1 \
  "gpt-5.1" \
  openai \
  0.00125 \
  0.01

Output:

Adding model to models.yml:
  Friendly name: gpt5-1
  API name: gpt-5.1
  Provider: openai
  Pricing: $0.00125 / $0.01 per 1K tokens

✓ Updated models.yml
✓ Validated YAML syntax
✓ Ready to test

scripts/verify_vertex_model.sh <model-name>

Check if a Gemini model is available in Vertex AI.

Usage:

# Check if Gemini 3 Pro is available
scripts/verify_vertex_model.sh gemini-3-pro-preview-11-2025

Output:

Checking Vertex AI for: gemini-3-pro-preview-11-2025
✓ GCP project: multivac-internal-prod
✓ Access token obtained
✗ Model not found (404)
Recommendation: Monitor for availability, check again in 1-2 weeks

scripts/run_test_benchmark.sh <model-name>

Run a small test benchmark to verify model works end-to-end.

Usage:

# Test GPT-5.1 with fizzbuzz benchmark
scripts/run_test_benchmark.sh gpt5-1

Output:

Running test benchmark: fizzbuzz
Model: gpt5-1
✓ Benchmark completed
✓ Result: PASS (100%)
✓ Tokens: 245 input, 89 output
✓ Cost: $0.002
Model is ready for production use

Workflow

1. Test API Access

First, verify you can call the model:

# Use test_model_access.sh
scripts/test_model_access.sh openai gpt-5.1

What to check:

• API key is set (OPENAI_API_KEY, ANTHROPIC_API_KEY, or gcloud auth)
• API call succeeds (not 401/403/404)
• Model returns expected structure
• Token usage is reported

For Gemini models:

• Uses Vertex AI (not public API)
• Requires gcloud auth application-default login
• Check availability with verify_vertex_model.sh

2. Find Model Information

Search for official documentation:

# Find API model name and pricing
scripts/find_model_info.sh "GPT-5.1 API documentation pricing"

What to gather:

• Exact API model name (e.g., gpt-5.1 not GPT-5.1)
• Provider (openai, anthropic, google)
• Input price per 1K tokens
• Output price per 1K tokens
• Context limits (if relevant)
• Special features (adaptive reasoning, caching, etc.)

Reference: See resources/provider_endpoints.md

3. Update models.yml

Add the model configuration:

# Add to models.yml
scripts/update_models_yml.sh \
  <friendly-name> \
  <api-name> \
  <provider> \
  <input-per-1k> \
  <output-per-1k>

Naming conventions:

• Friendly name: gpt5-1, claude-sonnet-4-5, gemini-3-pro
• API name: Exact string for API calls
• Use hyphens, lowercase

Also update:

• Model suites (benchmark_suite, extended_suite, dev_models)
• Add notes about special features
• Document agent CLI support (if available)

4. Run Test Benchmark

Verify end-to-end:

# Test with a simple benchmark
scripts/run_test_benchmark.sh <model-name>

What to verify:

• Benchmark completes successfully
• Results are reasonable (not garbage output)
• Token usage matches expectations
• Cost calculation works
• No errors in logs

5. Document the Model

Update relevant documentation:

• Add model to this skill's resource guide
• Note any special parameters (e.g., max_completion_tokens for GPT-5.1)
• Document authentication requirements
• Add to teaching prompts if needed

6. Optional: Run Full Eval

If model looks good:

# Run small eval suite
ailang eval-suite --models <model-name> --benchmarks fizzbuzz,recursion_factorial

# Run full suite (expensive!)
make eval-baseline EVAL_VERSION=vX.Y.Z FULL=true

Resources

Provider Endpoints

See resources/provider_endpoints.md for:

• API endpoint URLs for each provider
• Authentication methods
• How to test access manually
• Common errors and fixes

Pricing Guide

See resources/pricing_guide.md for:

• How to find official pricing
• Price conversion (per 1M → per 1K)
• Cost calculation verification
• Caching and discounts

Progressive Disclosure

This skill loads information progressively:

1. Always loaded: This SKILL.md file (workflow and script descriptions)
2. Execute as needed: Scripts in scripts/ (testing, updating, verification)
3. Load on demand: Resources (detailed endpoint docs, pricing references)

Notes

Important:

• Always test API access BEFORE updating models.yml
• Vertex AI (Gemini) requires gcloud auth, not API key
• GPT-5.1+ uses max_completion_tokens instead of max_tokens
• New models may not be available in all regions immediately
• Check for preview/beta status before adding to production suites

Prerequisites:

• API keys set in environment (OPENAI_API_KEY, ANTHROPIC_API_KEY)
• For Gemini: gcloud CLI installed and authenticated
• For Gemini: GCP project set (gcloud config set project PROJECT_ID)
• curl, python3, and jq available in PATH

Files modified by this skill:

• internal/eval_harness/models.yml - Model configurations
• (Optional) prompts/vX.Y.Z.md - Teaching prompts
• (Optional) .claude/skills/model-manager/resources/ - Local model database