Enhanced Document Metrics Implementation

Enhanced Document Metrics Implementation

Overview

Enhanced the Control Center document metrics to provide more meaningful and accurate information about user document usage, replacing simple upload counts with comprehensive metrics including date formatting, AI search storage calculations, and Azure Storage integration.

Issue Description

The original Control Center document metrics showed only basic upload counts without providing meaningful insights into:

• When documents were last uploaded (specific dates)
• AI search storage consumption based on document content
• Actual storage account usage for enhanced citations
• Proper date formatting for user-friendly display

Root Cause Analysis

The previous implementation:

1. Only counted daily uploads without showing specific dates
2. Did not calculate AI search storage size based on page content
3. Lacked integration with Azure Storage for actual file sizes
4. Did not provide the MM/DD/YYYY date format requested by users

Technical Implementation

Fixed in Version: 0.230.024

Backend Changes (route_backend_control_center.py)

Enhanced Document Metrics Structure

'document_metrics': {
    'personal_workspace_enabled': bool,
    'enhanced_citation_enabled': bool,
    'last_day_upload': 'MM/DD/YYYY or N/A',  # NEW: Date instead of count
    'total_documents': int,                   # Enhanced counting
    'ai_search_size': int,                   # NEW: pages × 80KB calculation
    'storage_account_size': int              # NEW: Azure Storage integration
}

Database Query Optimizations

Separate Queries to Avoid Cosmos DB MultipleAggregates Error:

1. Document Count Query:

SELECT VALUE COUNT(1) FROM c WHERE c.user_id = @user_id

2. Total Pages Query:

SELECT VALUE SUM(c.number_of_pages) FROM c WHERE c.user_id = @user_id

3. Most Recent Upload Date Query:

SELECT TOP 1 c.last_updated 
FROM c 
WHERE c.user_id = @user_id 
ORDER BY c.last_updated DESC

Date Formatting Logic

# Parse various date formats and convert to MM/DD/YYYY
if isinstance(last_updated, str):
    try:
        dt = datetime.fromisoformat(last_updated.replace('Z', '+00:00'))
    except:
        try:
            dt = datetime.strptime(last_updated, '%Y-%m-%d')
        except:
            dt = datetime.strptime(last_updated, '%Y-%m-%dT%H:%M:%S')
else:
    dt = last_updated
    
last_day_upload = dt.strftime('%m/%d/%Y')

AI Search Size Calculation

# Calculate AI search storage: pages × 80KB
total_pages = pages_result[0] if pages_result else 0
ai_search_size = total_pages * 80 * 1024  # 80KB per page in bytes

Azure Storage Integration

# Get actual file sizes when enhanced citations enabled
if enhanced_citations_enabled:
    try:
        storage_client = BlobServiceClient(account_url=storage_account_url, 
                                         credential=DefaultAzureCredential())
        container_client = storage_client.get_container_client(container_name)
        blob_list = container_client.list_blobs(name_starts_with=user_folder_prefix)
        
        total_size = sum(blob.size for blob in blob_list if blob.size)
    except Exception:
        # Fallback to estimated size
        total_size = ai_search_size

Frontend Changes (static/js/control-center.js)

Updated Document Metrics Rendering

function renderDocumentMetrics(user) {
    const metrics = user.activity?.document_metrics || {};
    
    return `
        <div class="metric-item">
            <div class="metric-label">Last Day:</div>
            <div class="metric-value">${metrics.last_day_upload || 'N/A'}</div>
        </div>
        <div class="metric-item">
            <div class="metric-label">Total Docs:</div>
            <div class="metric-value">${(metrics.total_documents || 0).toLocaleString()}</div>
        </div>
        <div class="metric-item">
            <div class="metric-label">AI Search:</div>
            <div class="metric-value">${formatBytes(metrics.ai_search_size || 0)}</div>
        </div>
        ${metrics.enhanced_citation_enabled ? `
        <div class="metric-item">
            <div class="metric-label">Storage:</div>
            <div class="metric-value">${formatBytes(metrics.storage_account_size || 0)}</div>
        </div>
        ` : ''}
    `;
}

Configuration Updates (config.py)

• Version updated to 0.230.024

Testing and Validation

Test Data Validation

Based on test user 07e61033-ea1a-4472-a1e7-6b9ac874984a:

• Total Documents: 33
• Total Pages: 2,619
• AI Search Size: 214,548,480 bytes (204.61 MB)
• Last Upload Date: 10/02/2025
• Enhanced Citations: Disabled

Functional Tests Created

1. test_document_metrics_implementation_verification.py - Comprehensive implementation verification
2. test_control_center_document_metrics_endpoint.py - API endpoint testing
3. test_document_metrics_database_queries.py - Database query validation

Validation Results

✅ Date Format: MM/DD/YYYY format correctly applied
✅ AI Search Size: Accurate calculation (pages × 80KB)
✅ Storage Integration: Azure Storage SDK properly integrated
✅ Database Queries: Separate queries avoid MultipleAggregates error
✅ Frontend Display: New format renders correctly
✅ Backward Compatibility: Existing functionality preserved

User Experience Improvements

Before

• Last Day: Simple upload count number
• No AI search storage information
• No actual storage account usage
• Generic numeric display

After

• Last Day: Specific date (MM/DD/YYYY) of most recent document upload
• AI Search: Calculated storage size based on document pages (pages × 80KB)
• Storage: Actual file sizes from Azure Storage when enhanced citations enabled
• User-Friendly: Formatted displays with proper units and comma separators

Impact Analysis

Performance Impact

• Positive: Separate queries reduce Cosmos DB MultipleAggregates errors
• Minimal: Additional queries are simple and efficient
• Optimized: Azure Storage calls only when enhanced citations enabled

User Benefits

1. Clearer Information: Specific dates instead of daily counts
2. Storage Awareness: Understanding of AI search storage consumption
3. Cost Transparency: Actual storage usage when enhanced citations enabled
4. Better Planning: Historical context with last upload dates

Deployment Notes

Prerequisites

• Azure Storage SDK properly configured
• Cosmos DB containers accessible
• Enhanced citations feature flag available

Rollback Plan

• Previous document metrics structure maintained in database
• Frontend can handle both old and new formats
• No breaking changes to existing APIs

Future Enhancements

Potential Improvements

1. Historical Trends: Track document upload patterns over time
2. Storage Optimization: Identify large files consuming excessive storage
3. Usage Analytics: Document access patterns and search frequency
4. Cost Projections: Estimate future storage costs based on usage trends

Monitoring Points

• Document metrics calculation performance
• Azure Storage API call frequency
• User satisfaction with new date format
• Control Center page load times with enhanced metrics

Related Issues

• Message count showing 0 (fixed with separate query approach)
• Control Center performance optimization
• Enhanced citations storage integration
• User activity metrics enhancement