Troubleshooting Gpt Oss And Vllm Errors

Troubleshooting gpt-oss and vLLM Errors

When to Use This Skill

Invoke this skill when you encounter:

• openai_harmony.HarmonyError messages in any context
• gpt-oss tool calling failures or unexpected behavior
• Token parsing errors with vLLM serving gpt-oss models
• Users asking about gpt-oss compatibility with frameworks like llama-stack

Critical First Step: Identify Error Source

IMPORTANT: openai_harmony.HarmonyError messages originate from the vLLM server, NOT from client applications (like llama-stack, LangChain, etc.).

Error Source Identification

1. Check the error origin:

   • If error contains openai_harmony.HarmonyError, it's from vLLM's serving layer
   • The client application is just reporting what vLLM returned
   • Do NOT search the client codebase for fixes

2. Correct investigation path:

   • Search vLLM GitHub issues and PRs
   • Check openai/harmony repository for parser issues
   • Review vLLM server configuration and startup flags
   • Examine HuggingFace model files (generation_config.json)

Common Error Patterns

Token Mismatch Errors

Error Pattern: Unexpected token X while expecting start token Y

Example: Unexpected token 12606 while expecting start token 200006

Meaning:

• vLLM expects special Harmony format control tokens
• Model is generating regular text tokens instead
• Token 12606 = "comment" (indicates model generating reasoning text instead of tool calls)

Known Issues:

• vLLM #22519: gpt-oss-20b tool_call token errors
• vLLM #22515: Same error, fixed by updating generation_config.json

Fixes:

1. Update model files from HuggingFace (see reference/model-updates.md)
2. Verify vLLM server flags for tool calling
3. Check generation_config.json EOS tokens

Tool Calling Not Working

Symptoms:

• Model describes tools in text but doesn't call them
• Empty tool_calls=[] arrays
• Tool responses in wrong format

Root Causes:

1. Missing vLLM server flags
2. Outdated model configuration files
3. Configuration mismatch between client and server

Configuration Requirements:

vLLM server must be started with:

--tool-call-parser openai --enable-auto-tool-choice

For demo tool server:

--tool-server demo

For MCP tool servers:

--tool-server ip-1:port-1,ip-2:port-2

Important: Only tool_choice='auto' is supported.

Investigation Workflow

1. Identify the error message:

   • Copy the exact error text
   • Note any token IDs mentioned

2. Search vLLM GitHub:

   • Use error text in issue search
   • Include "gpt-oss" and model size (20b/120b)
   • Check both open and closed issues

3. Check model configuration:

   • Verify generation_config.json is current
   • Compare against latest HuggingFace version
   • Look for recent commits that updated config

4. Review server configuration:

   • Check vLLM startup flags
   • Verify tool-call-parser settings
   • Confirm vLLM version compatibility

5. Check vLLM version:

   • Many tool calling issues resolved in recent vLLM releases
   • Update to latest version if encountering errors
   • Check vLLM changelog for gpt-oss-specific fixes

Quick Reference

Key Resources

• vLLM gpt-oss recipe: https://docs.vllm.ai/projects/recipes/en/latest/OpenAI/GPT-OSS.html
• Common issues: See reference/known-issues.md
• Model update procedure: See reference/model-updates.md

Diagnostic Commands

Check vLLM server health:

curl http://localhost:8000/health

List available models:

curl http://localhost:8000/v1/models

Check vLLM version:

pip show vllm

Progressive Disclosure

For detailed information:

• Known GitHub issues: See reference/known-issues.md
• Model file updates: See reference/model-updates.md
• Tool calling configuration: See reference/tool-calling-setup.md

Validation Steps

After implementing fixes:

1. Test simple tool calling with single function
2. Verify Harmony format tokens in responses
3. Check for token mismatch errors in logs
4. Test multi-turn conversations with tools
5. Monitor for "unexpected token" errors

If errors persist:

• Update vLLM to latest version
• Check vLLM GitHub for recent fixes and PRs
• Try different model variant (120b vs 20b)
• Review vLLM logs for additional error context

Converted and distributed by TomeVault — claim your Tome and manage your conversions.