# Spring AI Document Processing

> A complete guide for document processing, chunking strategies, ETL pipelines, and preparing data for RAG systems in Spring AI.

- Skill: `mat-garcia/spring-ai-document-processing` (Agent Skill)
- Install (CLI): `npx skillmds@latest add mat-garcia/spring-ai-document-processing`
- Raw SKILL.md: https://api.skillmd.com/api/skills/mat-garcia/spring-ai-document-processing/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- License: Complete terms in LICENSE.txt
- Author: mat-garcia (https://skillmd.com/u/mat-garcia)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/mat-garcia/spring-ai-document-processing

---


# Spring AI - Document Processing & ETL Pipelines

## Description

Complete guide for document processing, chunking strategies, ETL pipelines, and preparing data for RAG systems. Covers text splitting, metadata extraction, batch processing, and integration with vector stores.

## When to Use

- Preparing documents for RAG systems
- Splitting large documents
- Extracting metadata
- Batch processing files
- Data cleaning and normalization
- ETL workflows
- Knowledge base creation
- Document ingestion pipelines

## Topics Covered

### 1. Document Structure

- **Content**: Main text/body
- **Metadata**: Source, type, date, author, tags
- **ID**: Unique document identifier
- **Properties**: Custom attributes
- **Embeddings**: Associated vector representation

### 2. Text Splitting Strategies

#### Recursive Character Split

- Break on meaningful boundaries (paragraphs, sentences)
- Default strategy for most use cases
- Configurable separators
- Minimal content duplication
- Preserves semantic structure

#### Token-based Split

- Split based on token count
- Respects model's token limit
- Accurate for LLM processing
- Requires tokenizer
- Best for LLM input preparation

#### Sliding Window Split

- Fixed-size chunks with overlap
- Overlap prevents context loss
- Configurable stride
- Good for dense information

#### Hierarchical Split

- Split at multiple levels
- Document → Section → Subsection → Chunk
- Preserve document structure
- Support for queries at different levels

#### PDF-specific Split

- Extract text with formatting
- Handle tables, images, metadata
- Page break preservation
- Table of contents parsing

### 3. Chunking Parameters

- **Chunk size**: Typical 512-1024 tokens
- **Overlap**: 20-30% for continuity
- **Separator**: How to split (newline, paragraph)
- **Keep separator**: Preserve delimiters
- **Length function**: Custom sizing logic

### 4. Metadata Handling

- **Extraction**: Pull from document properties
- **Enrichment**: Add derived metadata
- **Preservation**: Keep throughout pipeline
- **Filtering**: Query by metadata
- **Updates**: Re-extract when needed

### 5. Document Processing Pipeline

```
Input Files → Parse → Split → Enrich →
Embed → Store → Index → Ready for RAG
```

### 6. ETL Workflow Patterns

- **Extract**: Parse source documents
- **Transform**: Clean, normalize, split
- **Load**: Store in vector DB
- **Monitor**: Track progress, errors
- **Reprocess**: Handle updates

## Code Patterns

### Basic Document Splitting

```java
@Service
public class DocumentProcessor {
    @Autowired
    private TextSplitter textSplitter;

    public List<Document> processDocument(String content) {
        return textSplitter.split(
            new Document(content, Map.of("source", "input.txt"))
        );
    }
}
```

### Configured Text Splitting

```java
@Configuration
public class TextSplitterConfig {
    @Bean
    public TextSplitter textSplitter() {
        return new RecursiveCharacterTextSplitter(
            1000,      // chunkSize
            200,       // overlapSize
            tokenCounter()
        );
    }

    @Bean
    public TokenCounter tokenCounter() {
        return new OpenAiTokenCounter();
    }
}
```

### File-based Document Processing

```java
@Service
public class FileDocumentService {
    @Autowired
    private TextSplitter splitter;

    @Autowired
    private VectorStore vectorStore;

    public void processPdfFile(File pdfFile) throws IOException {
        // Extract text
        String text = extractTextFromPdf(pdfFile);

        // Create document with metadata
        Document doc = new Document(text, Map.of(
            "source", pdfFile.getName(),
            "type", "pdf",
            "path", pdfFile.getAbsolutePath(),
            "lastModified", new Date(pdfFile.lastModified()).toString()
        ));

        // Split
        List<Document> chunks = splitter.split(doc);

        // Store
        vectorStore.add(chunks);
    }
}
```

### Batch Document Processing

```java
@Service
public class BatchDocumentProcessor {
    @Autowired
    private TextSplitter splitter;

    @Autowired
    private VectorStore vectorStore;

    public void processBatch(List<String> documentTexts) {
        List<Document> allChunks = documentTexts.stream()
            .map(text -> new Document(text))
            .flatMap(doc -> splitter.split(doc).stream())
            .toList();

        // Batch add to vector store
        vectorStore.add(allChunks);
    }
}
```

### Metadata Enrichment

```java
@Service
public class MetadataEnrichmentService {
    @Autowired
    private ChatClient chatClient;

    public Document enrichDocument(Document doc) {
        String content = doc.getContent();

        // Extract metadata using LLM
        Map<String, Object> metadata = new HashMap<>(doc.getMetadata());

        String summary = chatClient.prompt()
            .user("Summarize: " + content.substring(0, 500))
            .call()
            .content();

        String category = chatClient.prompt()
            .user("Categorize: " + content.substring(0, 500))
            .call()
            .content();

        metadata.put("summary", summary);
        metadata.put("category", category);

        return new Document(content, metadata);
    }
}
```

### ETL Pipeline

```java
@Service
public class DocumentEtlPipeline {
    @Autowired
    private FileDocumentService fileService;

    @Autowired
    private TextSplitter splitter;

    @Autowired
    private VectorStore vectorStore;

    @Autowired
    private EmbeddingModel embeddingModel;

    public void runEtl(String inputDirectory) throws IOException {
        // Extract phase
        List<File> files = listFiles(inputDirectory);
        logger.info("Found {} files to process", files.size());

        // Transform phase
        List<Document> chunks = new ArrayList<>();
        for (File file : files) {
            try {
                String content = readFile(file);
                Document doc = new Document(content,
                    extractMetadata(file));

                chunks.addAll(splitter.split(doc));
                logger.info("Processed: {}", file.getName());
            } catch (Exception e) {
                logger.error("Error processing {}: {}",
                    file.getName(), e.getMessage());
            }
        }

        logger.info("Created {} chunks", chunks.size());

        // Load phase - add to vector store
        vectorStore.add(chunks);
        logger.info("Loaded all chunks to vector store");
    }

    private Map<String, Object> extractMetadata(File file) {
        return Map.of(
            "source", file.getName(),
            "path", file.getAbsolutePath(),
            "size", file.length(),
            "lastModified", new Date(file.lastModified())
        );
    }
}
```

### Streaming Document Processing

```java
@Service
public class StreamingDocumentService {
    @Autowired
    private TextSplitter splitter;

    @Autowired
    private VectorStore vectorStore;

    public Flux<Document> processDocumentsStream(
            Flux<String> documentFlux) {

        return documentFlux
            .map(text -> new Document(text))
            .flatMap(doc -> Flux.fromIterable(splitter.split(doc)))
            .buffer(100)  // Batch for efficiency
            .doOnNext(batch -> vectorStore.add(batch))
            .flatMap(Flux::fromIterable);
    }
}
```

### Incremental Document Updates

```java
@Service
public class IncrementalUpdateService {
    @Autowired
    private VectorStore vectorStore;

    @Autowired
    private TextSplitter splitter;

    public void updateDocument(String docId, String newContent) {
        // Remove old chunks
        vectorStore.delete(
            vectorStore.similaritySearch(
                SearchRequest.query("*")
                    .withFilterExpression("source == '" + docId + "'")
            ).stream()
            .map(doc -> doc.getId())
            .toList()
        );

        // Add new chunks
        Document doc = new Document(newContent,
            Map.of("source", docId));
        vectorStore.add(splitter.split(doc));

        logger.info("Updated document: {}", docId);
    }
}
```

### Custom Text Splitter

```java
public class CustomTextSplitter implements TextSplitter {
    private final int chunkSize;
    private final int overlapSize;

    @Override
    public List<Document> split(Document document) {
        List<Document> chunks = new ArrayList<>();
        String text = document.getContent();

        int start = 0;
        while (start < text.length()) {
            int end = Math.min(start + chunkSize, text.length());

            // Find sentence boundary
            int actualEnd = text.lastIndexOf('.', end);
            if (actualEnd <= start) {
                actualEnd = end;
            }

            String chunk = text.substring(start, actualEnd).trim();

            Document chunkDoc = new Document(chunk,
                document.getMetadata());
            chunks.add(chunkDoc);

            start = actualEnd - overlapSize;
        }

        return chunks;
    }
}
```

## Configuration

### Text Splitter Configuration

```java
@Configuration
public class SplitterConfiguration {
    @Bean
    public TextSplitter recursiveSplitter() {
        return new RecursiveCharacterTextSplitter(
            1024,      // chunkSize
            256,       // overlapSize
            tokenCounter()
        );
    }

    @Bean
    public TextSplitter tokenSplitter() {
        return new TokenTextSplitter(
            512,       // tokensPerChunk
            100,       // tokensOverlap
            tokenCounter()
        );
    }

    @Bean
    public TokenCounter tokenCounter() {
        return new OpenAiTokenCounter();
    }
}
```

### Properties

```properties
# Document Processing
spring.ai.document.chunk-size=1024
spring.ai.document.overlap-size=200
spring.ai.document.separator=\n\n
spring.ai.document.preserve-separator=true

# ETL
spring.ai.etl.batch-size=100
spring.ai.etl.retry-attempts=3
spring.ai.etl.retry-delay=1s
spring.ai.etl.max-parallel-tasks=4
```

## Best Practices

- Use appropriate chunk sizes (512-1024 tokens)
- Include overlap to maintain context
- Preserve document structure metadata
- Implement error handling and recovery
- Monitor processing performance
- Clean and normalize text
- Test splitting with real data
- Cache splitter results
- Version your splitting strategy
- Log detailed metrics

## Related Skills

- `embeddings/SKILL.md` - Embedding documents
- `vector-stores/SKILL.md` - Storage
- `rag-retrieval/SKILL.md` - Using for RAG
- `batch-processing/SKILL.md` - Batch operations

## References

- API: `/pages/api/etl-pipeline.adoc`
- VectorDB: `/pages/api/vectordbs.adoc`
- Examples: Provider-specific document loading