Enhanced Vision Analysis Debug Logging

Enhanced Vision Analysis Debug Logging

Version: 0.233.202
Enhancement Type: Diagnostic Logging
Purpose: Diagnose GPT-5 vision analysis issues where no errors are thrown but results are incomplete

Problem

GPT-5 vision analysis was not throwing errors but wasn't working properly:

• Backend logs showed "Vision response not valid JSON, using raw text"
• GPT-4o worked correctly
• No detailed information about what was happening during the vision analysis process

Solution

Added comprehensive debug_print() logging throughout the analyze_image_with_vision_model() function to provide detailed visibility into every step of the vision analysis process.

Enhanced Logging Categories

1. Image Conversion & Preparation

[VISION_ANALYSIS] Image conversion for {document_id}:
  Image path: /path/to/image.png
  Original size: 8,340,622 bytes (7.95 MB)
  Base64 size: 11,120,828 characters
  MIME type: image/png

What to look for:

• Very large images might cause issues (> 20 MB)
• Incorrect MIME type detection
• Base64 encoding problems

2. Model Configuration

[VISION_ANALYSIS] Vision model selected: gpt-5
[VISION_ANALYSIS] Using APIM: False

What to look for:

• Correct model name
• APIM vs Direct connection method

3. Client Initialization

[VISION_ANALYSIS] Direct Azure OpenAI Configuration:
  Endpoint: https://your-resource.openai.azure.com/
  API Version: 2024-02-15-preview
  Auth Type: key

What to look for:

• Correct endpoint for the model deployment
• API version compatibility (vision requires 2024-02-15-preview or later)
• Authentication method matches configuration

4. API Parameter Selection

[VISION_ANALYSIS] Building API request parameters:
  Model (lowercase): gpt-5
  Uses max_completion_tokens: True
  Detection: o1=False, o3=False, gpt-5=True
  Token parameter: max_completion_tokens = 1000

What to look for:

• Correct detection of model type
• Proper parameter selection (max_completion_tokens for gpt-5, max_tokens for gpt-4o)
• Token limit appropriate for response

5. Request Details

[VISION_ANALYSIS] Sending request to Azure OpenAI...
  Message content types: text + image_url
  Image data URL prefix: data:image/png;base64,... (11120828 chars)

What to look for:

• Proper message structure
• Base64 data being sent

6. Response Metadata

[VISION_ANALYSIS] Response received from gpt-5
  Response ID: chatcmpl-ABC123XYZ
  Model used: gpt-5-2024-11-20
  Token usage: prompt=1245, completion=156, total=1401

What to look for:

• Actual model used (might differ from deployment name)
• Token usage patterns (high prompt tokens = large image)
• Response ID for tracking

7. Response Content Analysis

[VISION_ANALYSIS] Raw response received:
  Length: 823 characters
  First 500 chars: The image is a stylized promotional graphic...
  Last 100 chars: ...emphasizing the prestige and excitement of the event.
  Starts with JSON bracket: False
  Contains code fence: False

What to look for:

• CRITICAL: If Starts with JSON bracket: False, the response is NOT in JSON format
• If Contains code fence: True, response might be wrapped in markdown
• Response length (too short might indicate truncation)

8. JSON Parsing Attempt

[VISION_ANALYSIS] Attempting to clean JSON code fences...
  Cleaned length: 823 characters
  Cleaned first 200 chars: The image is a stylized promotional...
[VISION_ANALYSIS] Attempting to parse as JSON...
[VISION_ANALYSIS] ❌ JSON parsing failed!
  Error type: JSONDecodeError
  Error message: Expecting value: line 1 column 1 (char 0)
  Content that failed to parse (first 1000 chars): The image is a stylized...

What to look for:

• CRITICAL: If JSON parsing fails, shows WHY it failed
• Shows the exact content that couldn't be parsed
• Indicates if the model returned plain text instead of JSON

9. Successful JSON Parsing

[VISION_ANALYSIS] ✅ Successfully parsed JSON response!
  JSON keys: ['description', 'objects', 'text', 'analysis']

What to look for:

• All expected keys present: description, objects, text, analysis
• Missing keys indicate incomplete response

10. Final Analysis Structure

[VISION_ANALYSIS] Final analysis structure for {document_id}:
  Model: gpt-5
  Has 'description': True
  Has 'objects': True
  Has 'text': True
  Has 'analysis': True
  Description length: 234 chars
  Description preview: The image is a stylized promotional graphic...
  Objects count: 4
  Objects: ['jockeys', 'horses', 'artistic brushstrokes', 'text block']
  Text length: 523 chars
  Text preview: The 149th PREAKNESS May 18, 2024...

What to look for:

• All expected fields populated
• Reasonable content in each field
• Objects list populated (indicates vision working)
• Text extracted (indicates OCR working)

Diagnostic Workflow

When GPT-5 Shows "Vision response not valid JSON"

1. Check Response Format:

   Starts with JSON bracket: False  ← Problem!
   

   • If False, GPT-5 is returning plain text, not JSON
   • This is the most common issue

2. Check Response Content:

   First 500 chars: The image is a stylized promotional graphic...
   

   • Does it look like a description (plain text)?
   • Or does it look like JSON structure?

3. Check Token Usage:

   Token usage: prompt=15234, completion=89, total=15323
   

   • Very high prompt tokens (> 10k) = large image
   • Low completion tokens (< 100) might indicate truncated response

4. Check Model Version:

   Model used: gpt-5-2024-11-20
   

   • Might reveal model doesn't support JSON mode
   • Or model is preview version with different behavior

5. Check API Version:

   API Version: 2024-02-15-preview
   

   • Older API versions might not support certain features
   • Try newer version if available

Common Issues & Solutions

Issue 1: GPT-5 Returns Plain Text Instead of JSON

Symptoms:

Starts with JSON bracket: False
JSON parsing failed: Expecting value: line 1 column 1

Possible Causes:

1. GPT-5 doesn't support JSON mode - Some preview models don't support structured output
2. Prompt needs adjustment - Model not following JSON format instruction
3. Model interprets vision differently - Reasoning models might need different prompts

Solutions:

• Add response_format={"type": "json_object"} parameter (if supported)
• Modify prompt to be more explicit about JSON requirement
• Use post-processing to convert plain text to JSON structure

Issue 2: Large Image Causing Issues

Symptoms:

Original size: 8,340,622 bytes (7.95 MB)
Token usage: prompt=18945, completion=45, total=18990

Solutions:

• Image too large for context window
• Compress/resize image before analysis
• Split into multiple smaller images

Issue 3: Model Not Supporting Vision

Symptoms:

Error: Model does not support image inputs

Solutions:

• Verify model deployment supports vision
• Check API version is 2024-02-15-preview or later
• Confirm deployment region supports vision models

Testing GPT-5 vs GPT-4o

With enhanced logging, you can now compare:

GPT-4o Successful Response:

✅ Successfully parsed JSON response!
JSON keys: ['description', 'objects', 'text', 'analysis']
Objects count: 4

GPT-5 Problem Response:

❌ JSON parsing failed!
Starts with JSON bracket: False
Content that failed to parse: The image is a stylized promotional graphic...

This clearly shows GPT-5 is returning plain text descriptions instead of JSON structure.

Next Steps

If GPT-5 Returns Plain Text:

1. Modify Prompt for GPT-5 - Add stricter JSON formatting requirement
2. Enable JSON Mode - If model supports it: response_format={"type": "json_object"}
3. Post-Process Response - Parse plain text response into JSON structure
4. Use Different Approach - Some models prefer different instruction formats

If Model Doesn't Support JSON Mode:

Create fallback logic:

if 'gpt-5' in model_name and not response_is_json:
    # Parse natural language response into structured format
    vision_analysis = parse_natural_language_vision_response(content)

Files Modified

• functions_documents.py: Enhanced analyze_image_with_vision_model() with comprehensive logging
• config.py: Version updated to 0.233.202

Enabling Debug Output

Debug logging uses debug_print() which respects the application's debug settings:

1. Enable Debug Mode in application settings
2. Check Terminal Output where backend is running
3. Review Logs for [VISION_ANALYSIS] entries

All debug logs are prefixed with [VISION_ANALYSIS] for easy filtering.

References

• Vision Model Parameter Fix (v0.233.201)
• Multi-Modal Vision Analysis Feature (v0.229.088)
• Vision Model Detection Expansion (v0.229.089)