Model Debugging

Model Debugging Skill

Use this skill when:

• Investigating model failures, high error rates, or service issues
• Finding users affected by errors (402 billing, 403 permissions, 500 backend)
• Analyzing Tinybird/Cloudflare logs for patterns
• Diagnosing specific request failures

Related skill: Use tier-management to upgrade users or check balances after identifying issues here.

Understanding Model Monitor Error Rates

Why does the Model Monitor show high error rates when models work fine manually?

The Model Monitor at https://monitor.pollinations.ai shows all real-world traffic, including:

• 401 errors: Anonymous users without API keys (most common)
• 402 errors: Users with insufficient pollen balance or exhausted API key budget
• 403 errors: Users denied access to specific models (API key restrictions)
• 400 errors: Invalid request parameters (e.g., openai-audio without modalities param)
• 429 errors: Rate-limited requests
• 500/504 errors: Actual backend failures (investigate these)

When you test manually with a valid secret key (sk_), you bypass auth/quota issues, so models appear to work fine.

Key insight: High 401/402/403/400 rates are expected from real-world usage. Focus investigation on 500/504 errors.

Data Flow Architecture

User Request → enter.pollinations.ai (Cloudflare Worker)
                    ↓
              Logs to Cloudflare Workers Observability
                    ↓
              Events stored in D1 database
                    ↓
              Batched to Tinybird (async, 100-500 events)
                    ↓
              Model Monitor queries Tinybird (model_health.pipe)

Structured Logging: enter.pollinations.ai uses LogTape with:

• requestId: Unique per request (passed to downstream via x-request-id header)
• status, body: Full error response from downstream services
• Context: method, routePath, userAgent, ipAddress

Quick Diagnostics

1. Check Model Monitor

View current model health at: https://monitor.pollinations.ai

2. Query Recent Errors from D1 Database

# Via enter.pollinations.ai worker (requires wrangler)
cd enter.pollinations.ai
npx wrangler d1 execute pollinations-db --remote --command "SELECT model_requested, response_status, error_message, COUNT(*) as count FROM event WHERE response_status >= 400 AND created_at > datetime('now', '-1 hour') GROUP BY model_requested, response_status, error_message ORDER BY count DESC LIMIT 20"

3. Capture Live Logs

enter.pollinations.ai (Cloudflare Worker)

cd enter.pollinations.ai
wrangler tail --format json | tee logs.jsonl
# Or with formatting:
wrangler tail --format json | npx tsx scripts/format-logs.ts

image.pollinations.ai (EC2 systemd)

# Real-time logs
ssh enter-services "sudo journalctl -u image-pollinations.service -f"

# Last 3 minutes
ssh enter-services "sudo journalctl -u image-pollinations.service --since '3 minutes ago' --no-pager" > image-service-logs.txt

# Recent errors only
ssh enter-services "sudo journalctl -u image-pollinations.service -p err -n 50"

text.pollinations.ai (EC2 systemd)

# Real-time logs
ssh enter-services "sudo journalctl -u text-pollinations.service -f"

# Last 3 minutes
ssh enter-services "sudo journalctl -u text-pollinations.service --since '3 minutes ago' --no-pager" > text-service-logs.txt

Common Error Patterns

Azure Content Safety DNS Failure

Error: getaddrinfo ENOTFOUND gptimagemain1-resource.cognitiveservices.azure.com Cause: Azure Content Safety resource deleted or misconfigured Impact: Fail-open (content proceeds without safety check) Fix: Create new Azure Content Safety resource and update .env:

AZURE_CONTENT_SAFETY_ENDPOINT=https://<new-resource>.cognitiveservices.azure.com/
AZURE_CONTENT_SAFETY_API_KEY=<new-key>

Azure Kontext Content Filter

Error: Content rejected due to sexual/hate/violence content detection Cause: Azure's content moderation blocking prompts/images Impact: 400 error returned to user Fix: User error - prompt violates content policy

Vertex AI Invalid Image

Error: Provided image is not valid Cause: User passing unsupported image URL (e.g., Google Drive links) Impact: 400 error returned to user Fix: User error - need direct image URL

Translation Service Down

Error: No active translate servers available Cause: Translation service unavailable Impact: Prompts not translated (non-fatal) Fix: Check translation service status

OpenAI Audio Invalid Voice

Error: Invalid value for audio.voice Cause: User requesting unsupported voice name Impact: 400 error returned to user Fix: User error - use supported voices: alloy, echo, fable, onyx, nova, shimmer, coral, verse, ballad, ash, sage, etc.

Veo No Video Data

Error: No video data in response Cause: Vertex AI returned empty video response Impact: 500 error Fix: Check Vertex AI quota/status, may be transient

Environment Variables to Check

image.pollinations.ai

ssh enter-services "cat /home/ubuntu/pollinations/image.pollinations.ai/.env | grep -E 'AZURE|GOOGLE|CLOUDFLARE'"

Key variables:

• AZURE_CONTENT_SAFETY_ENDPOINT - Azure Content Safety API endpoint
• AZURE_CONTENT_SAFETY_API_KEY - Azure Content Safety API key
• GOOGLE_PROJECT_ID - Google Cloud project for Vertex AI
• AZURE_MYCELI_FLUX_KONTEXT_ENDPOINT - Azure Kontext model endpoint

text.pollinations.ai

ssh enter-services "cat /home/ubuntu/pollinations/text.pollinations.ai/.env | grep -E 'AZURE|OPENAI|GOOGLE'"

Updating Secrets

Secrets are stored encrypted with SOPS:

• image.pollinations.ai/secrets/env.json
• text.pollinations.ai/secrets/env.json

To update:

# Decrypt, edit, re-encrypt
sops image.pollinations.ai/secrets/env.json

# Deploy to server
sops --output-type dotenv -d image.pollinations.ai/secrets/env.json > /tmp/image.env
scp /tmp/image.env enter-services:/home/ubuntu/pollinations/image.pollinations.ai/.env
rm /tmp/image.env

# Restart service
ssh enter-services "sudo systemctl restart image-pollinations.service"

Log Analysis Commands

# Count errors by type
grep -i "error" image-service-logs.txt | grep -oE "(Azure Flux Kontext|Vertex AI|No active translate|getaddrinfo ENOTFOUND)" | sort | uniq -c | sort -rn

# Find content filter rejections
grep -i "Content rejected" image-service-logs.txt | sort | uniq -c

# Check DNS resolution on server
ssh enter-services "nslookup gptimagemain1-resource.cognitiveservices.azure.com"

Model-Specific Debugging

Cloudflare Workers Observability API

The enter.pollinations.ai worker has structured logging enabled. You can query logs programmatically via the Cloudflare Workers Observability API.

Prerequisites

1. Get Account ID

# From wrangler.toml
grep account_id enter.pollinations.ai/wrangler.toml

# Or from existing .env
grep CLOUDFLARE_ACCOUNT_ID image.pollinations.ai/.env

2. Create API Token with Workers Observability Permission

Via Cloudflare Dashboard:

1. Go to https://dash.cloudflare.com/profile/api-tokens
2. Click Create Token
3. Click Create Custom Token
4. Configure:
   • Token name: Workers Observability Read
   • Permissions:
     • Account → Workers Scripts → Read
     • Account → Workers Observability → Edit (required for query API)
   • Account Resources: Include → Your Account
5. Click Continue to summary → Create Token
6. Copy the token immediately (shown only once)

3. Store Token Securely

The token is stored in SOPS-encrypted secrets:

• Location: enter.pollinations.ai/secrets/env.json
• Key: CLOUDFLARE_OBSERVABILITY_TOKEN

To add/update:

# Step 1: Decrypt to temp file
cd /path/to/pollinations
sops -d enter.pollinations.ai/secrets/env.json > /tmp/env.json

# Step 2: Add the token (use jq)
jq '. + {"CLOUDFLARE_OBSERVABILITY_TOKEN": "your_token"}' /tmp/env.json > /tmp/env_updated.json

# Step 3: Re-encrypt (must rename to match .sops.yaml pattern)
cp /tmp/env_updated.json /tmp/env.json
sops -e /tmp/env.json > enter.pollinations.ai/secrets/env.json

# Step 4: Cleanup
rm /tmp/env.json /tmp/env_updated.json

# Verify
sops -d enter.pollinations.ai/secrets/env.json | jq 'keys'

Note: The .sops.yaml config requires filenames matching env.json$ pattern.

API Endpoint

POST https://api.cloudflare.com/client/v4/accounts/{account_id}/workers/observability/telemetry/query

Query Examples

Setup: Get Credentials from SOPS

# Extract credentials from encrypted secrets
ACCOUNT_ID=$(sops -d enter.pollinations.ai/secrets/env.json | jq -r '.CLOUDFLARE_ACCOUNT_ID')
API_TOKEN=$(sops -d enter.pollinations.ai/secrets/env.json | jq -r '.CLOUDFLARE_OBSERVABILITY_TOKEN')

List Available Log Keys (Working)

This endpoint works and shows what fields are available:

curl -s "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/workers/observability/telemetry/keys" \
  -H "Authorization: Bearer $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"timeframe": {"from": '$(( $(date +%s) - 86400 ))'000, "to": '$(date +%s)'000}, "datasets": ["workers"]}' | jq '.result[:10]'

Query Recent Errors (Last 15 Minutes)

Note: The /query endpoint requires a saved queryId. For ad-hoc queries, use the Cloudflare Dashboard Query Builder or wrangler tail.

# This format requires a saved query ID

# Query errors with status >= 400
curl -s "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/workers/observability/telemetry/query" \
  -H "Authorization: Bearer $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "timeframe": {
      "from": '$(( $(date +%s) - 900 ))'000,
      "to": '$(date +%s)'000
    },
    "parameters": {
      "datasets": ["workers"],
      "filters": [
        {"key": "$workers.scriptName", "operation": "eq", "type": "string", "value": "enter-pollinations-ai"},
        {"key": "$metadata.statusCode", "operation": "gte", "type": "number", "value": 400}
      ],
      "calculations": [{"operator": "count"}],
      "groupBys": [
        {"type": "string", "value": "$metadata.statusCode"},
        {"type": "string", "value": "$metadata.error"}
      ],
      "limit": 50
    }
  }' | jq '.result.events.events[:20]'

Query Errors by Model

curl -s "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/workers/observability/telemetry/query" \
  -H "Authorization: Bearer $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "timeframe": {
      "from": '$(( $(date +%s) - 3600 ))'000,
      "to": '$(date +%s)'000
    },
    "parameters": {
      "datasets": ["workers"],
      "filters": [
        {"key": "$workers.scriptName", "operation": "eq", "type": "string", "value": "enter-pollinations-ai"},
        {"key": "$metadata.statusCode", "operation": "gte", "type": "number", "value": 400}
      ],
      "calculations": [{"operator": "count"}],
      "groupBys": [
        {"type": "string", "value": "model"},
        {"type": "string", "value": "$metadata.statusCode"}
      ],
      "limit": 100
    }
  }' | jq '.result.calculations[0].aggregates'

Get Raw Error Events with Full Details

curl -s "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/workers/observability/telemetry/query" \
  -H "Authorization: Bearer $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "timeframe": {
      "from": '$(( $(date +%s) - 900 ))'000,
      "to": '$(date +%s)'000
    },
    "parameters": {
      "datasets": ["workers"],
      "filters": [
        {"key": "$workers.scriptName", "operation": "eq", "type": "string", "value": "enter-pollinations-ai"},
        {"key": "$metadata.statusCode", "operation": "gte", "type": "number", "value": 500}
      ],
      "limit": 20
    }
  }' | jq '.result.events.events[] | {
    timestamp: .timestamp,
    statusCode: ."$metadata".statusCode,
    error: ."$metadata".error,
    message: ."$metadata".message,
    requestId: ."$workers".requestId,
    url: ."$metadata".url
  }'

List Available Log Keys

curl -s "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/workers/observability/telemetry/keys" \
  -H "Authorization: Bearer $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "timeframe": {
      "from": '$(( $(date +%s) - 3600 ))'000,
      "to": '$(date +%s)'000
    },
    "datasets": ["workers"],
    "filters": [
      {"key": "$workers.scriptName", "operation": "eq", "type": "string", "value": "enter-pollinations-ai"}
    ]
  }' | jq '.result.keys'

Structured Logging in enter.pollinations.ai

The worker uses LogTape for structured logging with these key fields:

• requestId: Unique ID per request (first 8 chars shown in logs)
• method: HTTP method (GET, POST)
• routePath: Request URL
• status: Response status code
• duration: Request duration in ms

Downstream errors are logged with:

log.warn("Chat completions error {status}: {body}", {
    status: response.status,
    body: responseText,
});

Tinybird Analytics (Alternative)

For aggregated model health stats, query Tinybird directly:

# Get model health stats (last 5 minutes)
curl "https://api.europe-west2.gcp.tinybird.co/v0/pipes/model_health.json?token=$TINYBIRD_TOKEN" | jq '.data'

# Get detailed error breakdown
curl "https://api.europe-west2.gcp.tinybird.co/v0/pipes/model_errors.json?token=$TINYBIRD_TOKEN" | jq '.data'

The Tinybird token is a read-only public token found in:

• apps/model-monitor/src/hooks/useModelMonitor.js

Debugging Workflow

1. Check Model Monitor - https://monitor.pollinations.ai

   • Identify which models have high error rates
   • Note the error code breakdown (401, 402, 403, 400, 500, etc.)

2. Query Cloudflare Logs - Use the API queries above

   • Get raw error events with full details
   • Look for patterns in error messages

3. Correlate with Request ID - If you have a specific request ID:

   # Filter by request ID
   curl -s "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/workers/observability/telemetry/query" \
     -H "Authorization: Bearer $API_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{
       "timeframe": {"from": '$(( $(date +%s) - 86400 ))'000, "to": '$(date +%s)'000},
       "parameters": {
         "datasets": ["workers"],
         "filters": [
           {"key": "$workers.requestId", "operation": "eq", "type": "string", "value": "REQUEST_ID_HERE"}
         ],
         "limit": 100
       }
     }' | jq '.result.events.events'
   

4. Check Backend Logs - If error is from downstream service:

   # Image service
   ssh enter-services "sudo journalctl -u image-pollinations.service --since '5 minutes ago'"
   
   # Text service
   ssh enter-services "sudo journalctl -u text-pollinations.service --since '5 minutes ago'"
   

5. Test Model Directly - Verify if model is actually broken:

   TOKEN=$(grep ENTER_API_TOKEN_REMOTE enter.pollinations.ai/.testingtokens | cut -d= -f2)
   
   # Test text model
   curl -s 'https://gen.pollinations.ai/v1/chat/completions' \
     -H "Authorization: Bearer $TOKEN" \
     -H 'Content-Type: application/json' \
     -d '{"model": "MODEL_NAME", "messages": [{"role": "user", "content": "Test"}]}' \
     -w "\nHTTP: %{http_code}\n"
   
   # Test image model
   curl -s 'https://gen.pollinations.ai/image/test?model=MODEL_NAME&width=256&height=256' \
     -H "Authorization: Bearer $TOKEN" \
     -w "\nHTTP: %{http_code}\n" -o /dev/null
   

Current Status & Limitations

Cloudflare Observability API

What works:

• /telemetry/keys - List available log fields ✅
• /telemetry/values - Get unique values for a field ✅
• Token stored in SOPS: enter.pollinations.ai/secrets/env.json ✅

Limitations:

• /telemetry/query requires a saved queryId from the dashboard
• For ad-hoc queries, use Cloudflare Dashboard → Workers & Pages → pollinations-enter → Observability → Investigate
• Or use wrangler tail for real-time logs

Alternative: Tinybird (Recommended for Aggregates)

Tinybird provides pre-aggregated model health stats and raw event data.

Token Locations

1. Public read-only token (for pipes only): apps/model-monitor/src/hooks/useModelMonitor.js
2. Admin token (for raw SQL queries): enter.pollinations.ai/observability/.tinyb (in token field)

Basic Queries (Public Token)

# Public read-only token from apps/model-monitor
TINYBIRD_TOKEN="p.eyJ1IjogImFjYTYzZjc5LThjNTYtNDhlNC05NWJjLWEyYmFjMTY0NmJkMyIsICJpZCI6ICJmZTRjODM1Ni1iOTYwLTQ0ZTYtODE1Mi1kY2UwYjc0YzExNjQiLCAiaG9zdCI6ICJnY3AtZXVyb3BlLXdlc3QyIn0.Wc49vYoVYI_xd4JSsH_Fe8mJk7Oc9hx0IIldwc1a44g"

# Get model health (last 5 min)
curl -s "https://api.europe-west2.gcp.tinybird.co/v0/pipes/model_health.json?token=$TINYBIRD_TOKEN" | jq '.data'

Raw SQL Queries (Admin Token)

For querying the raw generation_event datasource, use the admin token from .tinyb:

# Get admin token from .tinyb file
TINYBIRD_ADMIN_TOKEN=$(jq -r '.token' enter.pollinations.ai/observability/.tinyb)

# Find users with frequent 403 errors (last 24 hours)
curl -s "https://api.europe-west2.gcp.tinybird.co/v0/sql?token=$TINYBIRD_ADMIN_TOKEN" \
  --data-urlencode "q=SELECT user_id, user_github_username, user_tier, count() as error_403_count 
FROM generation_event 
WHERE response_status = 403 
  AND start_time > now() - interval 24 hour 
  AND user_id != '' 
  AND user_id != 'undefined' 
GROUP BY user_id, user_github_username, user_tier 
ORDER BY error_403_count DESC 
LIMIT 20"

# Find users with 500 errors (actual backend issues)
curl -s "https://api.europe-west2.gcp.tinybird.co/v0/sql?token=$TINYBIRD_ADMIN_TOKEN" \
  --data-urlencode "q=SELECT user_github_username, model_requested, error_message, count() as error_count 
FROM generation_event 
WHERE response_status >= 500 
  AND start_time > now() - interval 24 hour 
GROUP BY user_github_username, model_requested, error_message 
ORDER BY error_count DESC 
LIMIT 20"

# Check specific user's recent errors
curl -s "https://api.europe-west2.gcp.tinybird.co/v0/sql?token=$TINYBIRD_ADMIN_TOKEN" \
  --data-urlencode "q=SELECT start_time, response_status, model_requested, error_message 
FROM generation_event 
WHERE user_github_username = 'USERNAME_HERE' 
  AND start_time > now() - interval 24 hour 
ORDER BY start_time DESC 
LIMIT 50"

Datasource Schema

The generation_event datasource is defined in enter.pollinations.ai/observability/datasources/generation_event.datasource and includes:

• user_id, user_github_username, user_tier
• response_status, error_message, error_response_code
• model_requested, model_used
• total_price, total_cost
• start_time, end_time, response_time

Scripts

Helper scripts for common debugging tasks. Run from repo root.

Find Users with 403 Errors (Quota Issues)

# Find users with >10 403 errors in last 24 hours
.claude/skills/model-debugging/scripts/find-403-users.sh 24 10

# Filter by tier (e.g., only spore users)
.claude/skills/model-debugging/scripts/find-403-users.sh 24 10 spore

Find 500 Errors (Backend Issues)

# Find 500+ errors grouped by user/model/message
.claude/skills/model-debugging/scripts/find-500-errors.sh 24

Check Specific User's Errors

# See a user's recent errors
.claude/skills/model-debugging/scripts/check-user-errors.sh superbrainai 24

Notes

• 401 errors: User authentication issues (no API key) - expected from anonymous traffic
• 402 errors: Pollen/billing issues (user ran out of credits or key budget) - expected
• 403 errors: Permission issues (model not allowed for API key) - expected
• 400 errors: Usually user input errors (bad prompts, invalid params) - expected
• 500 errors: Backend/infrastructure issues - investigate these
• 504 errors: Timeouts (model too slow or hung) - investigate these

Tested Models (All Working as of 2025-12-22)