tools-only PDF Processing Limits Analysis
Version Implemented: 0.229.001
Current Issues:
- Inconsistent Limits: We have a 2.25MB download limit but check against 500MB DI limit
- Arbitrary Character Limits: 225k character limit for summarization doesn't align with actual DI capabilities
- Overly Restrictive: Many legitimate PDFs (like the 1.5MB NIST PDF) barely fit or exceed our limits
Recommended Changes:
Option 1: Align with Azure DI Limits (Recommended)
- Download limit: Increase to match Azure DI (500MB for S0, 4MB for F0 tier)
- Remove arbitrary character-based processing limits
- Let Azure DI handle what it can process
- Use summarization only when extracted text exceeds token context limits
Option 2: Keep Conservative Limits
- Current approach but with better size calculations
- Clear messaging about why we limit downloads
- Faster processing for most use cases
Option 3: Dynamic Limits Based on Tier
- Detect Azure DI tier and adjust limits accordingly
- S0 tier: 500MB limit
- F0 tier: 4MB limit
Current Limits Breakdown:
Download Limits:
- PDFs: 75k chars × 30 = 2.25MB
- Other content: 75k chars × 2 = 150KB
Processing Limits:
- Summarization trigger: 75k chars × 3 = 225k chars
- Azure DI check: 500MB
- Max pages: 2,000 (Azure DI limit)
Real-World Examples:
- NIST PDF: 1.5MB - barely fits current limits
- Academic papers: Often 2-10MB
- Government documents: Can be 50-100MB
- Technical manuals: Often exceed current limits
Recommendation:
Use Option 1 - Align with Azure DI limits for better user experience while maintaining intelligent summarization based on actual token limits rather than arbitrary character counts.