Video Indexer Debug Logging Enhancement Fix

Video Indexer Debug Logging Enhancement Fix

Version: 0.229.058

Issue Description

Customers experiencing failures with Video Indexer API calls lacked sufficient logging to diagnose the root cause of issues. The existing logging was minimal and didn't provide enough detail to troubleshoot authentication problems, upload failures, processing issues, or configuration errors.

Root Cause Analysis

The video indexer implementation had limited logging that only provided basic success/failure messages without detailed error information, request/response data, or step-by-step processing details. This made it difficult to:

1. Identify authentication configuration issues
2. Troubleshoot upload failures
3. Debug processing polling problems
4. Understand deletion failures
5. Track chunk processing errors

Technical Solution

Enhanced Debug Logging Implementation

Added comprehensive debug_print statements throughout the video indexer workflow:

1. Authentication Function (functions_authentication.py)

• get_video_indexer_account_token: Added detailed logging for:
  • Token acquisition process start
  • Azure environment detection
  • ARM scope configuration
  • DefaultAzureCredential initialization
  • ARM token acquisition
  • Settings validation
  • ARM API request details (URL, body)
  • Response status and content analysis
  • Error handling with detailed HTTP response information

2. Video Processing (functions_documents.py)

• process_video_document: Enhanced with logging for:
  • Processing initialization with all parameters
  • Configuration validation and missing settings detection
  • Authentication token acquisition
  • Enhanced citations blob upload tracking

3. Video Upload Process

• Upload to Video Indexer: Added logging for:
  • Upload URL and parameters
  • File upload progress
  • Response status and content validation
  • Video ID extraction and validation
  • Document metadata updates

4. Processing Polling

• Index Polling: Enhanced with:
  • Polling URL and attempt tracking
  • Response status monitoring
  • Processing progress and state tracking
  • Error handling for rate limiting and timeouts
  • Maximum polling attempt protection

5. Insights Extraction

• Transcript and OCR Processing: Added logging for:
  • Insights extraction initiation
  • Transcript and OCR block counts
  • Context item processing
  • 30-second chunk processing details

6. Chunk Processing (save_video_chunk)

• Video Chunk Saving: Enhanced with:
  • Chunk parameter validation
  • Timestamp conversion tracking
  • Embedding generation monitoring
  • Document metadata retrieval
  • Chunk ID generation
  • Search index upload validation

7. Video Deletion (delete_document)

• Video Indexer Cleanup: Added logging for:
  • Video file detection
  • Configuration validation
  • Authentication for deletion
  • Video ID retrieval from metadata
  • Deletion request and response tracking

Debug Configuration Integration

All debug logging uses the existing debug_print function from functions_debug.py, which:

• Only logs when enable_debug_logging is enabled in settings
• Provides consistent formatting with "DEBUG:" prefix
• Handles configuration errors gracefully
• Allows selective debugging without performance impact

Files Modified

1. functions_authentication.py

   • Enhanced get_video_indexer_account_token with comprehensive debug logging

2. functions_documents.py

   • Enhanced process_video_document with detailed processing logging
   • Enhanced save_video_chunk with chunk processing logging
   • Enhanced delete_document with video deletion logging

3. config.py

   • Updated version from "0.229.040" to "0.229.041"

Debugging Benefits

Before Enhancement

[VIDEO] ARM token acquired
[VIDEO] UPLOAD OK, videoId=abc123
[VIDEO] Account token acquired (len=1234)

After Enhancement

DEBUG: [VIDEO INDEXER AUTH] Starting token acquisition for video_id: None
DEBUG: [VIDEO INDEXER AUTH] Azure environment: commercial
DEBUG: [VIDEO INDEXER AUTH] Using ARM scope: https://management.azure.com/.default
DEBUG: [VIDEO INDEXER AUTH] DefaultAzureCredential initialized successfully
DEBUG: [VIDEO INDEXER AUTH] ARM token acquired successfully (length: 2048)
DEBUG: [VIDEO INDEXER AUTH] Settings extracted - Subscription: sub-123, Resource Group: rg-test, Account: acc-test, API Version: 2021-11-10-preview
DEBUG: [VIDEO INDEXER AUTH] ARM API URL: https://management.azure.com/subscriptions/sub-123/resourceGroups/rg-test/providers/Microsoft.VideoIndexer/accounts/acc-test/generateAccessToken?api-version=2021-11-10-preview
DEBUG: [VIDEO INDEXER AUTH] Request body: {'permissionType': 'Contributor', 'scope': 'Account'}
DEBUG: [VIDEO INDEXER AUTH] ARM API response status: 200
DEBUG: [VIDEO INDEXER AUTH] ARM API response keys: ['accessToken', 'expiresIn']
DEBUG: [VIDEO INDEXER AUTH] Account token acquired successfully (length: 1234)

Error Troubleshooting Improvements

Authentication Errors

• Detailed ARM token acquisition steps
• Configuration validation with specific missing settings
• HTTP response status and error messages
• Environment-specific endpoint validation

Upload Errors

• File path and parameter validation
• Upload request details
• HTTP response analysis
• Video ID extraction validation

Processing Errors

• Polling attempt tracking with limits
• Rate limiting and timeout handling
• Processing state and progress monitoring
• Insights extraction validation

Deletion Errors

• Video file type detection
• Configuration availability checking
• Video ID metadata verification
• Deletion request and response tracking

Testing

Created comprehensive functional test (test_video_indexer_debug_logging_enhancement.py) that validates:

1. Authentication Debug Logging: Verifies all debug statements in video indexer authentication
2. Processing Debug Logging: Confirms detailed logging in video processing workflow
3. Chunk Processing Logging: Validates chunk saving debug statements
4. Deletion Debug Logging: Checks video deletion logging
5. Import Validation: Ensures proper debug_print imports
6. Error Handling: Verifies error scenarios have appropriate debug logging

Impact Assessment

Positive Impact

• Faster Issue Resolution: Detailed logs enable quick identification of failure points
• Proactive Monitoring: Better visibility into processing status and performance
• Configuration Validation: Clear indication of missing or incorrect settings
• Customer Support: Comprehensive logs for support ticket resolution

Performance Impact

• Minimal Overhead: Debug logging only active when enabled in settings
• Conditional Execution: Debug statements use efficient conditional checking
• No Production Impact: Debug logging disabled by default in production

Configuration

To enable enhanced video indexer debugging:

1. Enable debug logging in admin settings:

   {
     "enable_debug_logging": true
   }
   

2. Monitor application logs for detailed Video Indexer debug output

Customer Support Benefits

With this enhancement, customer support can now:

1. Quickly identify authentication issues with detailed ARM token and API response logs
2. Troubleshoot upload failures with comprehensive request/response information
3. Monitor processing progress with detailed polling and state information
4. Debug configuration problems with clear validation and error messages
5. Track chunk processing with individual chunk success/failure details

This significantly improves the ability to resolve Video Indexer issues and provides customers with faster resolution times.