Spring AI Document Processing

Spring AI - Document Processing & ETL Pipelines

Description

Complete guide for document processing, chunking strategies, ETL pipelines, and preparing data for RAG systems. Covers text splitting, metadata extraction, batch processing, and integration with vector stores.

When to Use

• Preparing documents for RAG systems
• Splitting large documents
• Extracting metadata
• Batch processing files
• Data cleaning and normalization
• ETL workflows
• Knowledge base creation
• Document ingestion pipelines

Topics Covered

1. Document Structure

• Content: Main text/body
• Metadata: Source, type, date, author, tags
• ID: Unique document identifier
• Properties: Custom attributes
• Embeddings: Associated vector representation

2. Text Splitting Strategies

Recursive Character Split

• Break on meaningful boundaries (paragraphs, sentences)
• Default strategy for most use cases
• Configurable separators
• Minimal content duplication
• Preserves semantic structure

Token-based Split

• Split based on token count
• Respects model's token limit
• Accurate for LLM processing
• Requires tokenizer
• Best for LLM input preparation

Sliding Window Split

• Fixed-size chunks with overlap
• Overlap prevents context loss
• Configurable stride
• Good for dense information

Hierarchical Split

• Split at multiple levels
• Document → Section → Subsection → Chunk
• Preserve document structure
• Support for queries at different levels

PDF-specific Split

• Extract text with formatting
• Handle tables, images, metadata
• Page break preservation
• Table of contents parsing

3. Chunking Parameters

• Chunk size: Typical 512-1024 tokens
• Overlap: 20-30% for continuity
• Separator: How to split (newline, paragraph)
• Keep separator: Preserve delimiters
• Length function: Custom sizing logic

4. Metadata Handling

• Extraction: Pull from document properties
• Enrichment: Add derived metadata
• Preservation: Keep throughout pipeline
• Filtering: Query by metadata
• Updates: Re-extract when needed

5. Document Processing Pipeline

Input Files → Parse → Split → Enrich →
Embed → Store → Index → Ready for RAG

6. ETL Workflow Patterns

• Extract: Parse source documents
• Transform: Clean, normalize, split
• Load: Store in vector DB
• Monitor: Track progress, errors
• Reprocess: Handle updates

Code Patterns

Basic Document Splitting

@Service
public class DocumentProcessor {
    @Autowired
    private TextSplitter textSplitter;

    public List<Document> processDocument(String content) {
        return textSplitter.split(
            new Document(content, Map.of("source", "input.txt"))
        );
    }
}

Configured Text Splitting

@Configuration
public class TextSplitterConfig {
    @Bean
    public TextSplitter textSplitter() {
        return new RecursiveCharacterTextSplitter(
            1000,      // chunkSize
            200,       // overlapSize
            tokenCounter()
        );
    }

    @Bean
    public TokenCounter tokenCounter() {
        return new OpenAiTokenCounter();
    }
}

File-based Document Processing

@Service
public class FileDocumentService {
    @Autowired
    private TextSplitter splitter;

    @Autowired
    private VectorStore vectorStore;

    public void processPdfFile(File pdfFile) throws IOException {
        // Extract text
        String text = extractTextFromPdf(pdfFile);

        // Create document with metadata
        Document doc = new Document(text, Map.of(
            "source", pdfFile.getName(),
            "type", "pdf",
            "path", pdfFile.getAbsolutePath(),
            "lastModified", new Date(pdfFile.lastModified()).toString()
        ));

        // Split
        List<Document> chunks = splitter.split(doc);

        // Store
        vectorStore.add(chunks);
    }
}

Batch Document Processing

@Service
public class BatchDocumentProcessor {
    @Autowired
    private TextSplitter splitter;

    @Autowired
    private VectorStore vectorStore;

    public void processBatch(List<String> documentTexts) {
        List<Document> allChunks = documentTexts.stream()
            .map(text -> new Document(text))
            .flatMap(doc -> splitter.split(doc).stream())
            .toList();

        // Batch add to vector store
        vectorStore.add(allChunks);
    }
}

Metadata Enrichment

@Service
public class MetadataEnrichmentService {
    @Autowired
    private ChatClient chatClient;

    public Document enrichDocument(Document doc) {
        String content = doc.getContent();

        // Extract metadata using LLM
        Map<String, Object> metadata = new HashMap<>(doc.getMetadata());

        String summary = chatClient.prompt()
            .user("Summarize: " + content.substring(0, 500))
            .call()
            .content();

        String category = chatClient.prompt()
            .user("Categorize: " + content.substring(0, 500))
            .call()
            .content();

        metadata.put("summary", summary);
        metadata.put("category", category);

        return new Document(content, metadata);
    }
}

ETL Pipeline

@Service
public class DocumentEtlPipeline {
    @Autowired
    private FileDocumentService fileService;

    @Autowired
    private TextSplitter splitter;

    @Autowired
    private VectorStore vectorStore;

    @Autowired
    private EmbeddingModel embeddingModel;

    public void runEtl(String inputDirectory) throws IOException {
        // Extract phase
        List<File> files = listFiles(inputDirectory);
        logger.info("Found {} files to process", files.size());

        // Transform phase
        List<Document> chunks = new ArrayList<>();
        for (File file : files) {
            try {
                String content = readFile(file);
                Document doc = new Document(content,
                    extractMetadata(file));

                chunks.addAll(splitter.split(doc));
                logger.info("Processed: {}", file.getName());
            } catch (Exception e) {
                logger.error("Error processing {}: {}",
                    file.getName(), e.getMessage());
            }
        }

        logger.info("Created {} chunks", chunks.size());

        // Load phase - add to vector store
        vectorStore.add(chunks);
        logger.info("Loaded all chunks to vector store");
    }

    private Map<String, Object> extractMetadata(File file) {
        return Map.of(
            "source", file.getName(),
            "path", file.getAbsolutePath(),
            "size", file.length(),
            "lastModified", new Date(file.lastModified())
        );
    }
}

Streaming Document Processing

@Service
public class StreamingDocumentService {
    @Autowired
    private TextSplitter splitter;

    @Autowired
    private VectorStore vectorStore;

    public Flux<Document> processDocumentsStream(
            Flux<String> documentFlux) {

        return documentFlux
            .map(text -> new Document(text))
            .flatMap(doc -> Flux.fromIterable(splitter.split(doc)))
            .buffer(100)  // Batch for efficiency
            .doOnNext(batch -> vectorStore.add(batch))
            .flatMap(Flux::fromIterable);
    }
}

Incremental Document Updates

@Service
public class IncrementalUpdateService {
    @Autowired
    private VectorStore vectorStore;

    @Autowired
    private TextSplitter splitter;

    public void updateDocument(String docId, String newContent) {
        // Remove old chunks
        vectorStore.delete(
            vectorStore.similaritySearch(
                SearchRequest.query("*")
                    .withFilterExpression("source == '" + docId + "'")
            ).stream()
            .map(doc -> doc.getId())
            .toList()
        );

        // Add new chunks
        Document doc = new Document(newContent,
            Map.of("source", docId));
        vectorStore.add(splitter.split(doc));

        logger.info("Updated document: {}", docId);
    }
}

Custom Text Splitter

public class CustomTextSplitter implements TextSplitter {
    private final int chunkSize;
    private final int overlapSize;

    @Override
    public List<Document> split(Document document) {
        List<Document> chunks = new ArrayList<>();
        String text = document.getContent();

        int start = 0;
        while (start < text.length()) {
            int end = Math.min(start + chunkSize, text.length());

            // Find sentence boundary
            int actualEnd = text.lastIndexOf('.', end);
            if (actualEnd <= start) {
                actualEnd = end;
            }

            String chunk = text.substring(start, actualEnd).trim();

            Document chunkDoc = new Document(chunk,
                document.getMetadata());
            chunks.add(chunkDoc);

            start = actualEnd - overlapSize;
        }

        return chunks;
    }
}

Configuration

Text Splitter Configuration

@Configuration
public class SplitterConfiguration {
    @Bean
    public TextSplitter recursiveSplitter() {
        return new RecursiveCharacterTextSplitter(
            1024,      // chunkSize
            256,       // overlapSize
            tokenCounter()
        );
    }

    @Bean
    public TextSplitter tokenSplitter() {
        return new TokenTextSplitter(
            512,       // tokensPerChunk
            100,       // tokensOverlap
            tokenCounter()
        );
    }

    @Bean
    public TokenCounter tokenCounter() {
        return new OpenAiTokenCounter();
    }
}

Properties

# Document Processing
spring.ai.document.chunk-size=1024
spring.ai.document.overlap-size=200
spring.ai.document.separator=\n\n
spring.ai.document.preserve-separator=true

# ETL
spring.ai.etl.batch-size=100
spring.ai.etl.retry-attempts=3
spring.ai.etl.retry-delay=1s
spring.ai.etl.max-parallel-tasks=4

Best Practices

• Use appropriate chunk sizes (512-1024 tokens)
• Include overlap to maintain context
• Preserve document structure metadata
• Implement error handling and recovery
• Monitor processing performance
• Clean and normalize text
• Test splitting with real data
• Cache splitter results
• Version your splitting strategy
• Log detailed metrics

Related Skills

• embeddings/SKILL.md - Embedding documents
• vector-stores/SKILL.md - Storage
• rag-retrieval/SKILL.md - Using for RAG
• batch-processing/SKILL.md - Batch operations

References

• API: /pages/api/etl-pipeline.adoc
• VectorDB: /pages/api/vectordbs.adoc
• Examples: Provider-specific document loading