# Improvements

> Roadmap and analysis of potential improvements, design trade-offs, and feature priorities. Use when planning new features, evaluating what to build next, or understanding why certain approaches were chosen or rejected. Use when this capability is needed.

- Skill: `tomevault-io/improvements` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add tomevault-io/improvements`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/improvements/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Product & Planning
- Author: tomevault-io (https://skillmd.com/u/tomevault-io)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/tomevault-io/improvements

---


# Improvement Roadmap

## Core Design Principle: Client-Side Intelligence

The fundamental architecture decision is that **semantic understanding lives in the MCP client (the AI), not in the server**. The server is a fast, precise, lexical retrieval engine. The AI compensates for what the server doesn't do.

This means:
- The AI generates synonym expansions via OR queries — no synonym files needed
- The AI handles multilingual query formulation — per-language stemmed shadow fields complement this (see candidate E)
- The AI iterates on results (search, read, refine) — no "smart" ranking needed
- The server stays simple, fast, dependency-light, and debuggable

**Any proposed improvement must be evaluated against this principle.** If the AI client can do it, the server shouldn't duplicate it.

---

## Current Strengths (Do Not Regress)

### Excellent — Competitive Advantages
1. **MCP-native architecture** — AI client as the semantic layer is genuinely more powerful than static synonym/stemming configuration
2. **Structured passage output** — `score`, `matchedTerms`, `termCoverage`, `position` per passage is optimized for LLM consumption
3. **Leading wildcard optimization** — `content_reversed` field for efficient `*vertrag`-style queries (German compound words)
4. **Incremental crawling** — 4-way reconciliation diff (DELETE/ADD/UPDATE/SKIP) is production-grade
5. **Operational polish** — Schema version management, auto-reindex, OS-native notifications, NRT adaptive refresh, MCP App admin UI, lock file recovery

### Solid — Good Foundation
6. **Tika extraction pipeline** — Thorough content normalization (HTML entities, URL encoding, NFKC, ligature expansion)
7. **Faceted search** — SortedSetDocValues facets with AI-guided drill-down workflow
8. **Crawler lifecycle** — Pause/resume, directory watching, batch processing, config persistence

---

## Improvement Candidates

### Tier 1: High Impact — Amplify the Existing Architecture

These improve the **AI client's ability to use the server effectively** without duplicating semantic logic.

#### 1. Structured Multi-Filter Support
**Status**: Done
**Effort**: Medium
**Impact**: High

Implemented `filters[]` array with operators (`eq`, `in`, `not`, `not_in`, `range`), DrillSideways faceting for faceted fields, ISO-8601 date parsing, `activeFilters` with `matchCount` in response, `dateFieldHints` in `getIndexStats`, and backward compatibility with legacy `filterField`/`filterValue`. Also addresses item 6 (Date-Friendly Query Parameters) via the `range` operator with ISO-8601 support.

#### 2. Index Observability Tools
**Status**: Done
**Effort**: Low-Medium
**Impact**: High

Implemented two read-only MCP tools for index vocabulary exploration:

**`suggestTerms`** — Prefix-based term completion with doc frequency sorting. Auto-lowercases prefix for analyzed fields (using `LOWERCASE_WILDCARD_FIELDS`). Cross-segment aggregation via HashMap merge. Point fields rejected with helpful error message.

**`getTopTerms`** — Full term enumeration sorted by frequency. Warning for fields with >100K unique terms. Works with both analyzed and StringFields.

Both tools: empty results for nonexistent fields (graceful), validation rejects `LONG_POINT_FIELDS`, new `IndexObservabilityIntegrationTest` with 20 tests covering prefix matching, case handling, limits, empty index, cross-segment aggregation.

#### 3. Document Chunking for Long Documents
**Status**: Not started
**Effort**: High
**Impact**: High

Currently each file = one Lucene document. The highlighter reads only 10,000 chars (`withMaxLength`), so passages from the second half of long PDFs are never found.

**Approach**: Split documents into overlapping chunks (e.g., ~2000 chars with 200-char overlap) at paragraph boundaries. Each chunk is a Lucene document linked to its parent by `file_path`. Search returns chunk-level passages; the AI sees which section of the document is relevant.

**Considerations**:
- Increases document count significantly (a 100-page PDF might produce 200+ chunks)
- Facets and metadata should be duplicated across chunks (or stored on a parent doc)
- `getDocumentDetails` would need to reconstruct the full document from chunks
- Schema version bump required
- This is also a prerequisite if vector search is ever added (embeddings need chunk-level granularity)

**Key files**: `DocumentIndexer.java`, `LuceneIndexService.search()`, `DocumentCrawlerService`

#### 4. Sort Options
**Status**: ✅ Done
**Effort**: Low
**Impact**: Medium-High

Implemented sorting capability for search results by metadata fields.

**What was implemented:**
- Added `sortBy` parameter: `_score` (default), `modified_date`, `created_date`, `file_size`
- Added `sortOrder` parameter: `asc` or `desc` (defaults: desc for dates/size, desc for score)
- Validation for invalid sort fields and orders
- Secondary sort by score for tie-breaking when sorting by metadata
- Comprehensive documentation in README.md and query-syntax resource

**Examples:**
```json
// Most recently modified
{"query": "contract", "sortBy": "modified_date", "sortOrder": "desc"}

// Oldest documents
{"query": "*", "sortBy": "created_date", "sortOrder": "asc"}

// Smallest files
{"query": "summary", "sortBy": "file_size", "sortOrder": "asc"}
```

**Key decisions:**
- Default sort remains by relevance score (no breaking changes)
- Always compute scores even when sorting by metadata (needed for highlighting and tie-breaking)
- Use `SortedNumericSortField` for numeric/date fields (DocValues)
- Clear error messages for invalid parameters

**Key files**: `SearchRequest.java`, `LuceneIndexService.search()`, README.md, query-syntax resource

### Tier 2: Medium Impact — Expand Capabilities

#### 5. "More Like This" / Similar Document Search
**Status**: Not started
**Effort**: Medium
**Impact**: Medium

Lucene has `MoreLikeThis` built in. A `findSimilar` tool that takes a `file_path` and returns related documents would be useful for the AI ("find documents similar to this contract").

**Key files**: New tool in `LuceneSearchTools.java`, new method in `LuceneIndexService.java`

#### 6. Date-Friendly Query Parameters
**Status**: Done (addressed by Structured Multi-Filter Support)
**Effort**: Low
**Impact**: Medium

Implemented as part of the `filters[]` array with `range` operator and ISO-8601 date parsing. The `getIndexStats` tool now also returns `dateFieldHints` with min/max dates.

#### 7. OCR Support for Scanned PDFs
**Status**: Not started
**Effort**: Medium-High
**Impact**: Medium (depends on user's document mix)

Tika supports Tesseract OCR. Scanned PDFs are a blind spot — the content extractor returns empty text. Even basic OCR would dramatically expand coverage for users with scanned documents.

**Considerations**:
- Tesseract must be installed on the host system (external dependency)
- OCR is slow — may need async processing or a separate queue
- Should be opt-in via configuration (`ocr-enabled: true`)
- Language hints from filename or metadata can improve OCR accuracy

**Key files**: `FileContentExtractor.java`, `application.yaml`

#### 8. Expanded File Format Support
**Status**: Done
**Effort**: Low
**Impact**: Low-Medium

Added default support for 8 new file formats (all handled natively by Tika 3.2.3, no new dependencies):
- `.eml`, `.msg` (emails)
- `.md`, `.rst` (markup)
- `.html`, `.htm` (web pages)
- `.rtf` (rich text)
- `.epub` (ebooks)

Includes include-patterns in `application.yaml` + `ApplicationConfig.java`, test document generators, parameterized extraction tests, and README documentation.

**Remaining**: `.csv` (spreadsheets as text) could still be added.

**Key files**: `application.yaml`, `ApplicationConfig.java`, `TestDocumentGenerator.java`, `FileContentExtractorTest.java`, README.md

#### 8.5. Markdown Bold Highlighting for Search Results
**Status**: ✅ Done
**Effort**: Low
**Impact**: Medium

Changed search result highlighting from HTML `<em>` tags to markdown `**bold**` syntax for proper rendering in Claude Desktop.

**What was implemented:**
- Updated `IndividualPassageFormatter` to use `**` instead of `<em>` tags
- Modified `extractMatchedTerms()` in `LuceneIndexService` to parse markdown bold markers
- Updated all documentation and test expectations
- Highlighted terms now render as bold in markdown-aware interfaces

**Rationale:**
- HTML tags show as literal text in Claude Desktop markdown rendering
- Markdown `**bold**` syntax renders properly as visual emphasis
- More LLM-friendly and conventional for markdown-based interfaces
- Aligns with how Claude Desktop displays content

**Key files:** `IndividualPassageFormatter.java`, `LuceneIndexService.java`, `Passage.java`, test files

**Key decisions:**
- Chose markdown over HTML for native Claude Desktop rendering
- Changed from Lucene's default `<b>` tags to markdown `**`
- Maintained all existing functionality (matched term extraction, coverage calculation)

#### 9. Query Profiling and Debugging Tool
**Status**: ✅ Done
**Effort**: High
**Impact**: High

Implemented `profileQuery` MCP tool with multi-level analysis optimized for LLM/human readability.

**What it provides**:
- **Level 1 (Fast, always)**: Query structure analysis, term statistics (IDF, rarity), cost estimates, query rewrites
- **Level 2 (Opt-in)**: Filter impact analysis with selectivity metrics (requires N+1 queries)
- **Level 3 (Opt-in)**: Document scoring explanations parsed from Lucene's Explanation API into version-independent semantic structures
- **Level 4 (Opt-in)**: Facet cost analysis

**Key decisions**:
- Made expensive operations opt-in (default: fast ~5-10ms analysis)
- Parsed Lucene's `Explanation.toString()` into structured DTOs instead of exposing raw format (version-independent)
- Added human-readable categorizations ("very common term", "high selectivity filter")
- Generated actionable optimization recommendations
- Focused on semantic structure over Lucene internals

**What it enables**:
- Understanding why queries return certain results
- Debugging scoring and ranking
- Identifying which terms/filters are most impactful
- Query optimization without deep Lucene knowledge
- LLMs can now help users tune their queries based on profiling data

**Files**: 15 new DTOs in `mcp/dto/`, updates to `LuceneSearchTools.java` and `LuceneIndexService.java`

**Known limitations** (from Lucene's Explanation API):
1. **Cannot explain non-matches**: Only explains documents that *did* match; can't debug why expected documents didn't appear
2. **Automaton internals opaque**: Wildcard/regex queries compile to finite automata; can't trace which substring matched `*vertrag*` in "Arbeitsvertrag"
3. **No passage-level explanation**: UnifiedHighlighter computes passage scores internally but doesn't expose explanation
4. **Filter impact requires measurement**: Must run queries incrementally to measure filter selectivity (Lucene's cost() API is insufficient)
5. **Cross-document comparison manual**: Each explanation is independent; requires custom logic to compare "why doc A > doc B"

**Future enhancements** (see Tier 3 below):
- "Why didn't this match?" tool (explainNonMatch)
- Query comparison tool (compareQueries)
- Passage-level scoring explanation
- Enhanced optimization suggestions

**Key files**: `ProfileQueryRequest.java`, `ProfileQueryResponse.java`, `QueryAnalysis.java`, `DocumentScoringExplanation.java`, and 11 other DTOs

#### 10. Context Pollution Reduction via MCP Resources
**Status**: ✅ Done
**Effort**: Medium
**Impact**: High

Implemented MCP Resources pattern to move verbose documentation out of tool descriptions, reducing initial context load by ~70%.

**What was implemented**:
- Shortened tool descriptions from ~3,000 to ~900 characters (-70%)
- Shortened parameter descriptions by ~50%
- Created two comprehensive MCP Resources:
  - `lucene://docs/query-syntax` - 350+ line Lucene query syntax guide
  - `lucene://docs/profiling-guide` - 150+ line profiling analysis guide
- LLM can access detailed docs on-demand via `Read` tool

**Pattern established**:
- **Tool descriptions**: What it does, when to use it, critical warnings, reference to resource
- **Parameter descriptions**: Type and purpose only
- **MCP Resources**: Complete syntax, examples, best practices, edge cases

**Benefits**:
- Reduced MCP handshake context by ~73%
- Maintained all essential information
- Better documentation organization
- LLM-friendly on-demand details

**Key files**: `LuceneSearchTools.java` (resource specifications and handlers), all DTO files (shortened descriptions)

**Future applications**:
- Create resources for crawler configuration guide
- Create resource for index field schema documentation
- Create resource for troubleshooting common issues
- Pattern should be used for all future complex tools

#### 10.5. Search Result Visualization Enhancements
**Status**: Not started
**Effort**: Medium
**Impact**: Medium-High

Enhance visual presentation of search results in Claude Desktop through rich markdown formatting and optional document previews.

**Proposed enhancements:**

**Tier 1 (High Value, Low Cost) - Implement First:**
1. **Rich Markdown Formatting** - Format search results with proper markdown structure:
   - Headers for document titles
   - Code blocks for file paths
   - Bold/italic for metadata
   - Structured result cards with visual hierarchy
   - Syntax-highlighted code snippets for code files

2. **Structured Result Cards** - Clear visual layout:
   ```
   ### 📄 Document Title
   **Path:** `/path/to/file.pdf`
   **Modified:** 2024-01-15 | **Size:** 2.3 MB

   **Preview:**
   The **budget** for Q4 2024 was exceeded...
   ```

3. **Better Snippets** - Enhanced passage formatting:
   - Already using markdown bold for matched terms ✅
   - Context around matches
   - Syntax highlighting for code

**Tier 2 (Medium Value, Medium Cost) - Optional:**
4. **File Type Icons** - Embed tiny base64 icons (5-10 common types, ~50 KB total)
5. **Search Statistics** - Formatted facet counts, distribution charts as text

**Tier 3 (High Value, High Cost) - Make Optional:**
6. **Document Thumbnails** - MCP supports base64-encoded images in responses
   - PDF first page preview
   - Image file thumbnails
   - Office document previews
   - **Constraint:** 1 MB maximum per image (MCP limit)
   - **Performance:** Generation during indexing vs. on-demand
   - **Configuration:** Make opt-in via `thumbnails.enabled: false` (default)

**MCP Support:**
MCP tool responses support multiple content types:
- `{"type": "text", "text": "..."}` - Markdown-formatted text
- `{"type": "image", "data": "base64...", "mimeType": "image/jpeg"}` - Images

**Rationale:**
- Claude Desktop renders markdown natively - leverage it
- Visual hierarchy improves result scanning
- Thumbnails provide immediate document recognition
- All enhancements are additive (no breaking changes)

**Key files:** `LuceneSearchTools.java`, new `ThumbnailGenerator.java` (if implementing thumbnails), `SearchResponse.java`

**Implementation priority:** Start with Tier 1 (zero cost, immediate value), add thumbnails only if users request it.

#### 11. Automatic Phrase Proximity Expansion
**Status**: ✅ Done
**Effort**: Low-Medium
**Impact**: Medium-High

Implemented automatic expansion of exact phrase queries to include proximity matching, improving recall while maintaining precision through differential scoring.

**What was implemented:**
- Created `ProximityExpandingQueryParser` extending Lucene's QueryParser
- Automatically expands multi-word exact phrases: `"Domain Design"` → `("Domain Design")^2.0 OR ("Domain Design"~3)`
- Exact matches score highest (2.0x boost), proximity matches score lower
- User-specified slop honored (no expansion if user already used ~N)
- Single-word phrases not expanded (no benefit)
- 10 unit tests + 7 integration tests
- All 359 tests pass (no regressions)

**Rationale:**
- Users/Claude don't need to know slop syntax
- Finds "Domain-driven Design" when searching "Domain Design"
- Exact matches still rank highest via boost (typically 2-3x higher score)
- Deterministic, predictable behavior
- No external dependencies (pure Lucene)
- Solves the original use case without adding Solr ComplexPhraseQueryParser

**Real-world example:**
```
Query: "Domain Design"
Results by score:
1. "Domain Design" (exact) - Score: 0.6981 ⭐⭐⭐
2. "Domain-driven Design" - Score: 0.1360 ⭐⭐
3. "Domain Effective Design" - Score: 0.1360 ⭐⭐
4. "Domain Very Effective Design" - Score: 0.0864 ⭐
```

**Configuration:**
- Default slop: 3 words (configurable via constructor)
- Default exact boost: 2.0x (configurable via constructor)
- Future: Could make configurable via application.yaml

**Key decisions:**
- Only multi-word phrases get expanded (single words have no benefit)
- User-specified slop always honored (backward compatible)
- Boost ensures exact matches always rank highest
- Slop of 3 balances recall (find variations) vs precision (avoid noise)

**Key files:** `ProximityExpandingQueryParser.java`, `LuceneIndexService.java`, `ProximityExpandingQueryParserTest.java`, `AutomaticPhraseExpansionIntegrationTest.java`

**Alternative considered:** Solr ComplexPhraseQueryParser - rejected due to Solr dependency, less control over scoring, and SpanQuery limitations with stopwords

---

## Missing Features & Gaps

### Critical Gaps (Should be added to Tier 1)

#### A. Index Backup & Restore
**Status**: Not implemented
**Priority**: High
**Impact**: Critical for production use

Currently no way to backup or restore the index. Users risk data loss on corruption or system failure.

**Proposed tools**:
- `backupIndex` - Create snapshot of index to specified location
- `restoreIndex` - Restore index from backup
- `listBackups` - List available backups with timestamps

**Considerations**:
- Index must be locked during backup
- Incremental backups vs full snapshots
- Backup verification/integrity checks
- Storage location configuration

**Key files**: New tools in `LuceneSearchTools.java`, new backup service

#### B. Duplicate Document Detection
**Status**: Not implemented
**Priority**: Medium-High
**Impact**: High

Index has `content_hash` field but no tools to find duplicate documents.

**Proposed tool**: `findDuplicates`
```json
{
  "method": "content_hash",  // or "fuzzy_content", "title_similarity"
  "threshold": 0.95,
  "groupBy": "content_hash"
}
```

**Returns groups of duplicate documents** with suggestions for which to keep/remove.

**Use cases**:
- Cleanup after indexing multiple document sources
- Detect near-duplicates (different versions of same document)
- Index optimization (remove redundant documents)

**Key files**: New tool, new analysis method in `LuceneIndexService.java`

### Tier 2 Additions (Medium Impact)

#### C. Batch/Bulk Operations
**Status**: Not implemented
**Priority**: Medium
**Impact**: Medium

No support for batch operations - each operation requires separate tool call.

**Proposed features**:
- `batchSearch` - Run multiple queries in one call, return combined results
- `batchProfileQuery` - Profile multiple queries for comparison
- `batchGetDocuments` - Retrieve multiple documents by file path

**Benefits**:
- Reduced MCP round-trips
- More efficient for comparative analysis
- Better for reporting/export scenarios

#### D. Export & Report Generation
**Status**: Not implemented
**Priority**: Medium
**Impact**: Medium

No way to export search results, profiling data, or statistics for external use.

**Proposed tool**: `exportResults`
```json
{
  "source": "last_search",  // or "profile_results", "index_stats"
  "format": "csv",          // or "json", "markdown"
  "fields": ["file_path", "score", "language"],
  "destination": "/path/to/export.csv"
}
```

**Use cases**:
- Generate reports for stakeholders
- Export for external analysis (Excel, BI tools)
- Archive search results

#### E. Multi-Language Snowball Stemming (Per-Language Shadow Fields)
**Status**: ✅ **DONE** — implemented in SCHEMA_VERSION 3
**Priority**: Medium-High
**Impact**: High (especially for German)

**Implemented approach**: Per-language lemma shadow fields using OpenNLP (replaced original Snowball stemming plan). All documents are indexed with both German and English lemma fields (`content_lemma_de`, `content_lemma_en`) regardless of detected language. At query time, weighted OR queries combine exact matching on `content` (boost 2.0) with lemmatized matching on language-specific fields (dynamic boost based on language distribution). Exact matches always rank highest. Highlighting stays on the unstemmed `content` field.

See [PIPELINE.md](../../PIPELINE.md) for complete analyzer chain documentation, token examples, and query pipeline details.

**Key implementation decisions**:
- Dynamic boost weights derived from index language distribution: `boost = 0.3 + 0.7 * (langCount / totalDocs)`
- Language distribution cached and refreshed on NRT searcher refresh
- If explicit `language eq "xx"` filter present, only that language's lemma field is included at boost 1.0
- Highlighting unaffected (uses unstemmed `content` field)
- Documents found only via lemma fields get fallback passages (no bold markers)

#### E1b. OpenNLP Lemmatizer Token Cleanup
**Status**: Done (SCHEMA_VERSION 8)
**Effort**: Low
**Impact**: Medium — eliminates index noise, improves term observability tools

**Problem**: OpenNLPTokenizer retains punctuation as separate tokens, and German UD-GSD lemmatizer produces compound lemmas for contractions (`im` → `in+der`).

**Solution**: Added `TypeTokenFilter` (drops punctuation) and `CompoundLemmaSplittingFilter` (splits on `+`) to the lemmatizer chain.

See [PIPELINE.md](../../PIPELINE.md) for complete analyzer chain details and examples.

#### E2. Irregular Verb Stemming (Extends Snowball Stemming)
**Status**: Superseded — OpenNLP lemmatization handles irregular verbs correctly
**Priority**: N/A
**Impact**: N/A (problem solved by lemmatizer)
**Depends on**: Candidate E (Snowball stemming — replaced by OpenNLP lemmatization)

**Problem**: Snowball is an algorithmic suffix-stripper. It handles regular morphology well but fails on irregular forms that involve vowel changes (Ablaut), suppletion, or prefix patterns:

*English gaps:*
- `ran` → "ran" vs `run`/`running` → "run" (vowel change)
- `went` → "went" vs `go`/`going` → "go" (suppletion)
- `saw` → "saw" vs `see`/`seen` → "see" (vowel change)
- `wrote` → "wrote" vs `write`/`written` → "write"
- `analysis` → "analysi" vs `analyses` → "analys" (confirmed in P/R tests — different stems)

*German gaps (more severe due to Ablaut + ge- prefix):*
- `ging` → "ging" vs `gehen`/`gegangen` → different stems
- `fuhr` → "fuhr" vs `fahren`/`gefahren` → different stems
- `sprach` → "sprach" vs `sprechen`/`gesprochen` → different stems
- `war` → "war" vs `sein`/`gewesen` → completely unrelated forms (suppletion)
- `lief` → "lief" vs `laufen`/`gelaufen` → different stems

**Current P/R state**: P=0.975, R=1.000, F1=0.975 on the 45-doc test corpus. R=1.000 because the test corpus is small and queries are carefully chosen. In a real corpus, verb-based searches like "who ran the project" or "was entschieden wurde" would hit these gaps.

---

**Approach A: Hunspell Dictionary Stemming**

Use Lucene's built-in `HunspellStemFilter` with `.dic`/`.aff` dictionary files. Dictionary-based stemming correctly maps irregular forms to their lemma because it uses lookup tables rather than suffix rules.

*Implementation*: Additional shadow fields using HunspellStemFilter. See [PIPELINE.md](../../PIPELINE.md) for analyzer chain details.

*Pros:*
- Handles all irregulars via dictionary lookup — no manual maintenance of verb lists
- Well-maintained dictionaries exist (LibreOffice/SCOWL communities)
- Lucene-native — `HunspellStemFilter` is in `lucene-analysis-common` (already a dependency)
- Covers not just irregular verbs but also irregular noun plurals, adjective inflections, etc.
- Would also improve Snowball's edge cases (e.g., `analysis`/`analyses`)

*Cons:*
- **License problem for German**: The de_DE Hunspell dictionary ([igerman98](https://www.j3e.de/ispell/igerman98/)) is **GPL-2.0 OR GPL-3.0**. Per the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html), GPL is **Category X — prohibited** for bundling in Apache 2.0 projects. This applies to both code and data files.
- English dictionary (SCOWL) is **MIT + BSD** — Apache Category A, fully compatible.
- Slower than Snowball (dictionary lookups vs rule application)
- Dictionary files add ~5-10 MB per language
- Can produce multiple stems per token (ambiguity: "saw" → "see" AND "saw")
- Dictionary updates need tracking

*License workaround options:*
1. **User-provided dictionaries**: Ship without dictionaries, load from config path (`hunspell-dictionaries-path` in application.yaml). User downloads and places dictionaries themselves. GPL applies to the dictionary files, not to code that reads them.
2. **English only**: Bundle only the MIT/BSD English dictionary; German stays Snowball-only.
3. **Wait for re-licensing**: Monitor igerman98 — if it ever moves to LGPL or more permissive license.

*If user-provided dictionaries approach:*
```yaml
# application.yaml
hunspell:
  enabled: false  # opt-in
  dictionaries-path: ~/.mcplucene/dictionaries/
  # User places de_DE.dic, de_DE.aff, en_US.dic, en_US.aff there
```

---

**Approach B: StemmerOverrideFilter (Explicit Mapping)**

Lucene's `StemmerOverrideFilter` runs before Snowball and provides a hard override map. Only irregular forms need entries; regular forms fall through to Snowball unchanged.

*Implementation*: Insert `StemmerOverrideFilter` into the existing `StemmedUnicodeNormalizingAnalyzer` chain, before `SnowballFilter`:

```
StandardTokenizer → LowerCaseFilter → ICUFoldingFilter → StemmerOverrideFilter(map) → SnowballFilter
```

*Override map examples:*
```java
// English (~200 irregular verbs, ~5-6 forms each ≈ ~1000 entries)
"ran" → "run", "went" → "go", "gone" → "go",
"saw" → "see", "seen" → "see",
"wrote" → "write", "written" → "write",
"paid" → "pay", "analyses" → "analysis", ...

// German (~170 strong/mixed verbs, ~5-6 forms each ≈ ~1000 entries)
"ging" → "gehen", "gegangen" → "gehen",
"fuhr" → "fahren", "gefahren" → "fahren",
"sprach" → "sprechen", "gesprochen" → "sprechen",
"war" → "sein", "gewesen" → "sein",
"lief" → "laufen", "gelaufen" → "laufen", ...
```

*Pros:*
- **No license issues** — override maps are our own code, Apache 2.0
- Precise — no false conflations beyond what we explicitly define
- Composable — sits in front of existing Snowball chain, no new fields needed
- Tiny overhead (HashMap lookup per token)
- No new dependencies
- Easy to test — each override is a deterministic mapping

*Cons:*
- Manual maintenance of override maps (~2000 entries total for DE+EN)
- Incomplete coverage — only catches forms we enumerate
- Doesn't scale to new languages without per-language work
- The "long tail" of irregulars is large: English ~200 verbs × ~5 forms, German ~170 verbs × ~6 forms
- Doesn't cover irregular noun plurals or adjective forms (only verbs, unless we add those too)
- Override map targets must match Snowball's output stem for regular words (e.g., "went" → "go", but Snowball stems "go" to "go", "goes" to "goe" — we'd need "went" → "go" AND update "goes" → "go" to normalize Snowball's irregular output)

*Key consideration*: The override target must be the Snowball stem of the base form, not the dictionary lemma. E.g., if Snowball stems "house" → "hous", then override "houses" → "hous" (not "house"). Since `StemmerOverrideFilter` runs BEFORE `SnowballFilter`, overridden tokens skip Snowball entirely — so the target must be the final desired stem. This means the override map is Snowball-version-dependent.

---

**Approach C: OpenNLP Lemmatizer (Lucene-integrated, Apache 2.0)** — IMPLEMENTED

Lucene's `OpenNLPLemmatizerFilter` provides true lemmatization via trained models. This approach was chosen and implemented.

*Implementation*: Shadow fields `content_lemma_de` / `content_lemma_en` using OpenNLP pipeline with sentence detection, POS tagging, and lemmatization. See [PIPELINE.md](../../PIPELINE.md) for complete analyzer chain documentation and token examples.

**Critical difference from Snowball**: Requires complete analyzer chain with OpenNLPTokenizer (sentence detection + tokenization) + POS tagging before lemmatization.

*Two modes available:*
1. **Dictionary-only**: `DictionaryLemmatizer` — HashMap lookup from `word[tab]postag[tab]lemma` file. Fast (O(1) per token), but requires POS tags and a comprehensive dictionary file.
2. **MaxEnt model**: `LemmatizerME` — statistical model (Maximum Entropy). Handles unseen words. Can be combined with dictionary (dictionary tried first, model for OOV fallback).

*Available models (all Apache 2.0):*

| Model | Source | Size | Notes |
|-------|--------|------|-------|
| EN lemmatizer | [OpenNLP models](https://opennlp.apache.org/models.html) | ~1-5 MB (est.) | `opennlp-en-ud-ewt-lemmas-1.3-2.5.4.bin` |
| EN POS tagger | OpenNLP models | ~5-15 MB (est.) | `opennlp-en-ud-ewt-pos-1.3-2.5.4.bin` |
| EN sentence detector | OpenNLP models | ~1 MB (est.) | `opennlp-en-ud-ewt-sentence-1.3-2.5.4.bin` |
| EN tokenizer | OpenNLP models | ~1 MB (est.) | `opennlp-en-ud-ewt-tokens-1.3-2.5.4.bin` |
| DE lemmatizer (small) | [DE-Lemma](https://github.com/mawiesne/DE-Lemma) | 861 KB | UD-GSD based, Apache 2.0 |
| DE lemmatizer (large) | DE-Lemma | 14 MB | UD-HDT based, Apache 2.0 |
| DE lemmatizer (huge) | DE-Lemma | 131 MB | Wikipedia-trained (36.1M sentences), Apache 2.0 |
| DE POS tagger | OpenNLP models | ~5-15 MB (est.) | `opennlp-de-ud-gsd-pos-1.3-2.5.4.bin` |
| DE sentence detector | OpenNLP models | ~1 MB (est.) | `opennlp-de-ud-gsd-sentence-1.3-2.5.4.bin` |
| DE tokenizer | OpenNLP models | ~1 MB (est.) | `opennlp-de-ud-gsd-tokens-1.3-2.5.4.bin` |

*Memory and performance analysis:*

**Startup impact:**
- Each language requires 4 models loaded into memory: sentence detector, tokenizer, POS tagger, lemmatizer
- Estimated heap per language: ~20-50 MB for small/medium models, ~150+ MB for DE Wikipedia model
- Models are loaded once at startup and shared across threads (MaxentModel is thread-safe)
- Current server startup: ~2 seconds. Model loading could add 1-3 seconds per language
- **Total memory impact**: ~40-100 MB additional heap for both languages with medium-sized models. This is significant for a server that currently runs lean (~50-100 MB heap)

**Index-time impact (critical path):**
- POS tagging is the bottleneck: MaxEnt model inference per token, not just a lookup
- Estimated throughput: ~10,000-50,000 tokens/second (vs ~500,000+ for Snowball)
- For a 10-page PDF (~5,000 tokens): Snowball ~10ms, OpenNLP ~100-500ms
- **Full reindex** of 10,000 documents: could add 15-60 minutes to crawl time
- POS tagging accuracy affects lemmatization: wrong POS tag → wrong lemma

**Query-time impact:**
- Query strings are short (typically 1-5 tokens) — OpenNLP pipeline overhead is negligible per query
- But the analyzer must be invoked for each stemmed field query (same as current Snowball approach)
- Estimated: ~1-5ms additional per query (acceptable)

**Index size impact:**
- Lemmatized tokens are typically shorter than or equal to surface forms (same as stemming)
- No significant additional index size vs Snowball shadow fields

*Pros:*
- **All Apache 2.0** — library, Lucene integration, EN models, DE models (DE-Lemma)
- True lemmatization: handles ALL irregulars (verbs, nouns, adjectives) correctly
- Handles unseen words via MaxEnt model (not limited to dictionary entries)
- German models trained on 36.1M sentences (Wikipedia) — extensive vocabulary coverage
- Lucene-native `TokenFilter` — same integration pattern as Snowball
- Dictionary + model hybrid: fast lookup for known words, statistical fallback for unknown

*Cons:*
- **Heavy pipeline**: requires sentence detector + tokenizer + POS tagger + lemmatizer (4 models per language vs 0 for Snowball)
- **Memory overhead**: ~40-100 MB additional heap for both languages (doubles current footprint)
- **Index-time performance**: 10-50x slower than Snowball per token due to POS tagging
- **Startup time**: +1-3 seconds for model loading
- **New dependency**: `lucene-analysis-opennlp` module + model files bundled or downloaded
- **Model files**: must be bundled in JAR or downloaded on first run (~20-60 MB total for medium models)
- **POS accuracy dependency**: lemmatization quality depends on POS tagging accuracy. Wrong POS → wrong lemma (e.g., "saw" tagged as NN → "saw", tagged as VBD → "see")
- **Complexity**: separate complete analyzer (can't just add a filter to existing chain), language detection must route to correct analyzer
- Contradicts "fast, simple server" principle more than other approaches

---

**Comparison**

| Aspect | Hunspell | StemmerOverrideFilter | OpenNLP Lemmatizer |
|--------|----------|----------------------|--------------------|
| Irregular verb coverage | Complete (dictionary) | ~90% of common verbs (manual) | Complete (trained model) |
| Irregular noun/adj coverage | Yes | Only if manually added | Yes |
| German license | **GPL — blocked** | Apache 2.0 (our code) | **Apache 2.0** |
| English license | MIT/BSD — OK | Apache 2.0 (our code) | **Apache 2.0** |
| Maintenance burden | Dictionary updates (community) | Manual map curation | Model updates (community) |
| Performance (index time) | Medium (dict lookup) | Minimal (HashMap) | **Slow (10-50x vs Snowball)** |
| Performance (query time) | Fast | Fastest | Fast (short queries) |
| Memory overhead | ~5-10 MB/lang (dict files) | None | **~20-50 MB/lang (models)** |
| Startup impact | Minimal | None | **+1-3 seconds** |
| New dependencies | Dictionary files only | None | `lucene-analysis-opennlp` + 4 models/lang |
| False conflations | Possible (ambiguous words) | Only what we define | POS-dependent (mostly correct) |
| Complexity | Low (single filter) | Low (single filter) | **High (full NLP pipeline)** |
| Effort | Medium | Medium | High |

**Recommendation**: Start with **Approach B (StemmerOverrideFilter)** for both languages — no license issues, no dependencies, minimal performance impact, precise control. Cover the ~50 most common irregular verbs per language first (~90% of practical recall gap). The AI client handles the remaining long tail through query expansion.

**If broader coverage is needed later**: **Approach C (OpenNLP)** is the strongest fully-Apache-2.0 option for complete irregular form handling. The memory and performance costs are significant but may be acceptable if the server handles large corpora where irregular verb recall matters. Consider making it opt-in via configuration (`lemmatizer.enabled: true`, `lemmatizer.engine: opennlp`) with model files loaded on demand.

**English-only Hunspell (Approach A)** remains viable for English specifically (MIT/BSD license), with user-provided dictionaries for German as a power-user option.

**Client-side complementary approach**: Regardless of server-side choice, the AI client can always expand irregular forms via OR queries ("ran OR run OR running"). This is already the approach for synonyms and works well for the irregular verbs that server-side stemming doesn't cover.

---

#### OpenNLP as a Platform Investment: NLP-vs-LLM Boundary Analysis

**Note:** This analysis was written for server deployment scenarios. For the **desktop/Claude Desktop use case**, most OpenNLP features beyond basic lemmatization are rejected due to resource constraints (see "Explicitly Rejected" section). The architectural principle remains valid, but the desktop context shifts the boundary heavily toward client-side LLM processing.

If the OpenNLP pipeline were committed to in a **server deployment** (not the current desktop use case), it becomes a **platform investment** — not just "better stemming" but a foundation for multiple features. The key insight: paying the startup/memory cost once unlocks capabilities beyond lemmatization.

**The Principle: "NLP enriches the index, LLM enriches the query"**

NLP processing at index time adds **structure to unstructured data** — this structure lives permanently in the index and benefits every future query. LLM processing at query time adds **intelligence from context** — understanding user intent, expanding queries semantically, iterating on results. These are complementary, not competing.

The boundary criterion: **If a capability produces deterministic, cacheable, document-level structure, it belongs in the NLP layer. If it requires contextual reasoning, user intent, or cross-document synthesis, it belongs in the LLM layer.**

---

**Feature 2: Sentence Detection for Document Chunking**

OpenNLP's `SentenceDetectorME` identifies sentence boundaries — a critical primitive for document chunking (Tier 1 candidate #3).

*Why it matters:*
- Current chunking would split at arbitrary character boundaries → sentences cut mid-word
- Sentence-aware chunking produces semantically coherent passages
- Better chunks → better search passages → more useful results for the AI client

*Why NLP, not LLM:*
- Sentence detection is a well-solved NLP problem (>99% accuracy for EN/DE)
- Deterministic, fast, runs once at index time
- LLM-based sentence splitting would be absurdly expensive for every document
- The model is already loaded for the lemmatization pipeline (zero marginal cost)

*Synergy with existing chunking candidate:*
- Chunking at sentence boundaries with overlap → each chunk is 5-10 sentences
- Paragraphs detected via whitespace + sentence boundaries
- Chunk metadata includes sentence count, position in document

*Impact assessment:*
- **Value**: High (prerequisite for quality chunking)
- **Effort**: Near-zero if OpenNLP pipeline exists (sentence model already loaded)
- **Memory**: 0 additional (already loaded for lemmatizer tokenizer)
- **Index time**: Included in tokenizer pass

---

**Feature 3: POS-Based Field Indexing**

With POS tags available, index only specific parts of speech into specialized fields.

*Examples:*
- `content_nouns` — only nouns, for concept-level search
- `content_verbs` — only verbs, for action-level search

*Assessment: Marginal value*
- The AI client can already focus searches via query formulation
- BM25 naturally weights content-bearing terms (nouns) higher than function words
- Additional fields increase index size without clear user benefit
- The search interface has no natural way to express "search only nouns"

*Verdict:* Not recommended. The complexity-to-benefit ratio is too low. If specific use cases emerge (e.g., domain-specific terminology extraction), reconsider.

---

**Feature 4: Noun Phrase Extraction**

Extract multi-word noun phrases (e.g., "Arbeitsvertrag", "supply chain management") as atomic units.

*Potential value:*
- Index "supply chain management" as a single term → exact phrase matching without proximity operators
- Extract compound nouns that are written as separate words in English but would be single words in German

*Assessment: Moderate value, but overlaps with LLM capabilities*
- The AI client already handles phrase queries ("supply chain management")
- OpenNLP chunker models have moderate accuracy for complex phrases
- German compound nouns are already single tokens (no extraction needed)
- English multi-word terms benefit, but the AI can formulate phrase queries

*Verdict:* Interesting but not high-priority. Could add value for automated keyword extraction / tag generation at index time.

---

**NLP-vs-LLM Boundary Summary**

| Capability | Server-Side (NLP) | Client-Side (LLM) | Winner |
|-----------|-------------------|-------------------|--------|
| **Lemmatization** | Index-time, deterministic, all docs once | Per-query OR expansion ("ran OR run") | **NLP** — scales better |
| **NER** | Index-time extraction → facets | Read each doc, extract on demand | **NLP** — pre-extract

…(truncated)