Praveen7287 Document Preprocessing Instructions
Step-by-Step Process
- Detect file format from the file extension
- Extract text content using appropriate library:
- PDF: Use PyPDF2 with fallback for scanned documents
- DOCX: Use python-docx preserving headings and structure
- TXT/MD: Read with UTF-8 encoding
- Clean the text by removing excessive whitespace and special characters
- Split into chunks using paragraph boundaries first, then sentence boundaries
- Add metadata to each chunk including source file and position
Implementation Details
def process_document(file_path):
# Implementation here
pass