Improvements

Improvement Roadmap

Core Design Principle: Client-Side Intelligence

The fundamental architecture decision is that semantic understanding lives in the MCP client (the AI), not in the server. The server is a fast, precise, lexical retrieval engine. The AI compensates for what the server doesn't do.

This means:

• The AI generates synonym expansions via OR queries — no synonym files needed
• The AI handles multilingual query formulation — per-language stemmed shadow fields complement this (see candidate E)
• The AI iterates on results (search, read, refine) — no "smart" ranking needed
• The server stays simple, fast, dependency-light, and debuggable

Any proposed improvement must be evaluated against this principle. If the AI client can do it, the server shouldn't duplicate it.

Current Strengths (Do Not Regress)

Excellent — Competitive Advantages

1. MCP-native architecture — AI client as the semantic layer is genuinely more powerful than static synonym/stemming configuration
2. Structured passage output — score, matchedTerms, termCoverage, position per passage is optimized for LLM consumption
3. Leading wildcard optimization — content_reversed field for efficient *vertrag-style queries (German compound words)
4. Incremental crawling — 4-way reconciliation diff (DELETE/ADD/UPDATE/SKIP) is production-grade
5. Operational polish — Schema version management, auto-reindex, OS-native notifications, NRT adaptive refresh, MCP App admin UI, lock file recovery

Solid — Good Foundation

6. Tika extraction pipeline — Thorough content normalization (HTML entities, URL encoding, NFKC, ligature expansion)
7. Faceted search — SortedSetDocValues facets with AI-guided drill-down workflow
8. Crawler lifecycle — Pause/resume, directory watching, batch processing, config persistence

Improvement Candidates

Tier 1: High Impact — Amplify the Existing Architecture

These improve the AI client's ability to use the server effectively without duplicating semantic logic.

1. Structured Multi-Filter Support

Status: Done Effort: Medium Impact: High

Implemented filters[] array with operators (eq, in, not, not_in, range), DrillSideways faceting for faceted fields, ISO-8601 date parsing, activeFilters with matchCount in response, dateFieldHints in getIndexStats, and backward compatibility with legacy filterField/filterValue. Also addresses item 6 (Date-Friendly Query Parameters) via the range operator with ISO-8601 support.

2. Index Observability Tools

Status: Done Effort: Low-Medium Impact: High

Implemented two read-only MCP tools for index vocabulary exploration:

suggestTerms — Prefix-based term completion with doc frequency sorting. Auto-lowercases prefix for analyzed fields (using LOWERCASE_WILDCARD_FIELDS). Cross-segment aggregation via HashMap merge. Point fields rejected with helpful error message.

getTopTerms — Full term enumeration sorted by frequency. Warning for fields with >100K unique terms. Works with both analyzed and StringFields.

Both tools: empty results for nonexistent fields (graceful), validation rejects LONG_POINT_FIELDS, new IndexObservabilityIntegrationTest with 20 tests covering prefix matching, case handling, limits, empty index, cross-segment aggregation.

3. Document Chunking for Long Documents

Status: Not started Effort: High Impact: High

Currently each file = one Lucene document. The highlighter reads only 10,000 chars (withMaxLength), so passages from the second half of long PDFs are never found.

Approach: Split documents into overlapping chunks (e.g., ~2000 chars with 200-char overlap) at paragraph boundaries. Each chunk is a Lucene document linked to its parent by file_path. Search returns chunk-level passages; the AI sees which section of the document is relevant.

Considerations:

• Increases document count significantly (a 100-page PDF might produce 200+ chunks)
• Facets and metadata should be duplicated across chunks (or stored on a parent doc)
• getDocumentDetails would need to reconstruct the full document from chunks
• Schema version bump required
• This is also a prerequisite if vector search is ever added (embeddings need chunk-level granularity)

Key files: DocumentIndexer.java, LuceneIndexService.search(), DocumentCrawlerService

4. Sort Options

Status: ✅ Done Effort: Low Impact: Medium-High

Implemented sorting capability for search results by metadata fields.

What was implemented:

• Added sortBy parameter: _score (default), modified_date, created_date, file_size
• Added sortOrder parameter: asc or desc (defaults: desc for dates/size, desc for score)
• Validation for invalid sort fields and orders
• Secondary sort by score for tie-breaking when sorting by metadata
• Comprehensive documentation in README.md and query-syntax resource

Examples:

// Most recently modified
{"query": "contract", "sortBy": "modified_date", "sortOrder": "desc"}

// Oldest documents
{"query": "*", "sortBy": "created_date", "sortOrder": "asc"}

// Smallest files
{"query": "summary", "sortBy": "file_size", "sortOrder": "asc"}

Key decisions:

• Default sort remains by relevance score (no breaking changes)
• Always compute scores even when sorting by metadata (needed for highlighting and tie-breaking)
• Use SortedNumericSortField for numeric/date fields (DocValues)
• Clear error messages for invalid parameters

Key files: SearchRequest.java, LuceneIndexService.search(), README.md, query-syntax resource

Tier 2: Medium Impact — Expand Capabilities

5. "More Like This" / Similar Document Search

Status: Not started Effort: Medium Impact: Medium

Lucene has MoreLikeThis built in. A findSimilar tool that takes a file_path and returns related documents would be useful for the AI ("find documents similar to this contract").

Key files: New tool in LuceneSearchTools.java, new method in LuceneIndexService.java

6. Date-Friendly Query Parameters

Status: Done (addressed by Structured Multi-Filter Support) Effort: Low Impact: Medium

Implemented as part of the filters[] array with range operator and ISO-8601 date parsing. The getIndexStats tool now also returns dateFieldHints with min/max dates.

7. OCR Support for Scanned PDFs

Status: Not started Effort: Medium-High Impact: Medium (depends on user's document mix)

Tika supports Tesseract OCR. Scanned PDFs are a blind spot — the content extractor returns empty text. Even basic OCR would dramatically expand coverage for users with scanned documents.

Considerations:

• Tesseract must be installed on the host system (external dependency)
• OCR is slow — may need async processing or a separate queue
• Should be opt-in via configuration (ocr-enabled: true)
• Language hints from filename or metadata can improve OCR accuracy

Key files: FileContentExtractor.java, application.yaml

8. Expanded File Format Support

Status: Done Effort: Low Impact: Low-Medium

Added default support for 8 new file formats (all handled natively by Tika 3.2.3, no new dependencies):

• .eml, .msg (emails)
• .md, .rst (markup)
• .html, .htm (web pages)
• .rtf (rich text)
• .epub (ebooks)

Includes include-patterns in application.yaml + ApplicationConfig.java, test document generators, parameterized extraction tests, and README documentation.

Remaining: .csv (spreadsheets as text) could still be added.

Key files: application.yaml, ApplicationConfig.java, TestDocumentGenerator.java, FileContentExtractorTest.java, README.md

8.5. Markdown Bold Highlighting for Search Results

Status: ✅ Done Effort: Low Impact: Medium

Changed search result highlighting from HTML <em> tags to markdown **bold** syntax for proper rendering in Claude Desktop.

What was implemented:

• Updated IndividualPassageFormatter to use ** instead of <em> tags
• Modified extractMatchedTerms() in LuceneIndexService to parse markdown bold markers
• Updated all documentation and test expectations
• Highlighted terms now render as bold in markdown-aware interfaces

Rationale:

• HTML tags show as literal text in Claude Desktop markdown rendering
• Markdown **bold** syntax renders properly as visual emphasis
• More LLM-friendly and conventional for markdown-based interfaces
• Aligns with how Claude Desktop displays content

Key files: IndividualPassageFormatter.java, LuceneIndexService.java, Passage.java, test files

Key decisions:

• Chose markdown over HTML for native Claude Desktop rendering
• Changed from Lucene's default <b> tags to markdown **
• Maintained all existing functionality (matched term extraction, coverage calculation)

9. Query Profiling and Debugging Tool

Status: ✅ Done Effort: High Impact: High

Implemented profileQuery MCP tool with multi-level analysis optimized for LLM/human readability.

What it provides:

• Level 1 (Fast, always): Query structure analysis, term statistics (IDF, rarity), cost estimates, query rewrites
• Level 2 (Opt-in): Filter impact analysis with selectivity metrics (requires N+1 queries)
• Level 3 (Opt-in): Document scoring explanations parsed from Lucene's Explanation API into version-independent semantic structures
• Level 4 (Opt-in): Facet cost analysis

Key decisions:

• Made expensive operations opt-in (default: fast ~5-10ms analysis)
• Parsed Lucene's Explanation.toString() into structured DTOs instead of exposing raw format (version-independent)
• Added human-readable categorizations ("very common term", "high selectivity filter")
• Generated actionable optimization recommendations
• Focused on semantic structure over Lucene internals

What it enables:

• Understanding why queries return certain results
• Debugging scoring and ranking
• Identifying which terms/filters are most impactful
• Query optimization without deep Lucene knowledge
• LLMs can now help users tune their queries based on profiling data

Files: 15 new DTOs in mcp/dto/, updates to LuceneSearchTools.java and LuceneIndexService.java

Known limitations (from Lucene's Explanation API):

1. Cannot explain non-matches: Only explains documents that did match; can't debug why expected documents didn't appear
2. Automaton internals opaque: Wildcard/regex queries compile to finite automata; can't trace which substring matched *vertrag* in "Arbeitsvertrag"
3. No passage-level explanation: UnifiedHighlighter computes passage scores internally but doesn't expose explanation
4. Filter impact requires measurement: Must run queries incrementally to measure filter selectivity (Lucene's cost() API is insufficient)
5. Cross-document comparison manual: Each explanation is independent; requires custom logic to compare "why doc A > doc B"

Future enhancements (see Tier 3 below):

• "Why didn't this match?" tool (explainNonMatch)
• Query comparison tool (compareQueries)
• Passage-level scoring explanation
• Enhanced optimization suggestions

Key files: ProfileQueryRequest.java, ProfileQueryResponse.java, QueryAnalysis.java, DocumentScoringExplanation.java, and 11 other DTOs

10. Context Pollution Reduction via MCP Resources

Status: ✅ Done Effort: Medium Impact: High

Implemented MCP Resources pattern to move verbose documentation out of tool descriptions, reducing initial context load by ~70%.

What was implemented:

• Shortened tool descriptions from ~3,000 to ~900 characters (-70%)
• Shortened parameter descriptions by ~50%
• Created two comprehensive MCP Resources:
  • lucene://docs/query-syntax - 350+ line Lucene query syntax guide
  • lucene://docs/profiling-guide - 150+ line profiling analysis guide
• LLM can access detailed docs on-demand via Read tool

Pattern established:

• Tool descriptions: What it does, when to use it, critical warnings, reference to resource
• Parameter descriptions: Type and purpose only
• MCP Resources: Complete syntax, examples, best practices, edge cases

Benefits:

• Reduced MCP handshake context by ~73%
• Maintained all essential information
• Better documentation organization
• LLM-friendly on-demand details

Key files: LuceneSearchTools.java (resource specifications and handlers), all DTO files (shortened descriptions)

Future applications:

• Create resources for crawler configuration guide
• Create resource for index field schema documentation
• Create resource for troubleshooting common issues
• Pattern should be used for all future complex tools

10.5. Search Result Visualization Enhancements

Status: Not started Effort: Medium Impact: Medium-High

Enhance visual presentation of search results in Claude Desktop through rich markdown formatting and optional document previews.

Proposed enhancements:

Tier 1 (High Value, Low Cost) - Implement First:

1. Rich Markdown Formatting - Format search results with proper markdown structure:

   • Headers for document titles
   • Code blocks for file paths
   • Bold/italic for metadata
   • Structured result cards with visual hierarchy
   • Syntax-highlighted code snippets for code files

2. Structured Result Cards - Clear visual layout:

   ### 📄 Document Title
   **Path:** `/path/to/file.pdf`
   **Modified:** 2024-01-15 | **Size:** 2.3 MB
   
   **Preview:**
   The **budget** for Q4 2024 was exceeded...
   

3. Better Snippets - Enhanced passage formatting:

   • Already using markdown bold for matched terms ✅
   • Context around matches
   • Syntax highlighting for code

Tier 2 (Medium Value, Medium Cost) - Optional: 4. File Type Icons - Embed tiny base64 icons (5-10 common types, ~50 KB total) 5. Search Statistics - Formatted facet counts, distribution charts as text

Tier 3 (High Value, High Cost) - Make Optional: 6. Document Thumbnails - MCP supports base64-encoded images in responses

• PDF first page preview
• Image file thumbnails
• Office document previews
• Constraint: 1 MB maximum per image (MCP limit)
• Performance: Generation during indexing vs. on-demand
• Configuration: Make opt-in via thumbnails.enabled: false (default)

MCP Support: MCP tool responses support multiple content types:

• {"type": "text", "text": "..."} - Markdown-formatted text
• {"type": "image", "data": "base64...", "mimeType": "image/jpeg"} - Images

Rationale:

• Claude Desktop renders markdown natively - leverage it
• Visual hierarchy improves result scanning
• Thumbnails provide immediate document recognition
• All enhancements are additive (no breaking changes)

Key files: LuceneSearchTools.java, new ThumbnailGenerator.java (if implementing thumbnails), SearchResponse.java

Implementation priority: Start with Tier 1 (zero cost, immediate value), add thumbnails only if users request it.

11. Automatic Phrase Proximity Expansion

Status: ✅ Done Effort: Low-Medium Impact: Medium-High

Implemented automatic expansion of exact phrase queries to include proximity matching, improving recall while maintaining precision through differential scoring.

What was implemented:

• Created ProximityExpandingQueryParser extending Lucene's QueryParser
• Automatically expands multi-word exact phrases: "Domain Design" → ("Domain Design")^2.0 OR ("Domain Design"~3)
• Exact matches score highest (2.0x boost), proximity matches score lower
• User-specified slop honored (no expansion if user already used ~N)
• Single-word phrases not expanded (no benefit)
• 10 unit tests + 7 integration tests
• All 359 tests pass (no regressions)

Rationale:

• Users/Claude don't need to know slop syntax
• Finds "Domain-driven Design" when searching "Domain Design"
• Exact matches still rank highest via boost (typically 2-3x higher score)
• Deterministic, predictable behavior
• No external dependencies (pure Lucene)
• Solves the original use case without adding Solr ComplexPhraseQueryParser

Real-world example:

Query: "Domain Design"
Results by score:
1. "Domain Design" (exact) - Score: 0.6981 ⭐⭐⭐
2. "Domain-driven Design" - Score: 0.1360 ⭐⭐
3. "Domain Effective Design" - Score: 0.1360 ⭐⭐
4. "Domain Very Effective Design" - Score: 0.0864 ⭐

Configuration:

• Default slop: 3 words (configurable via constructor)
• Default exact boost: 2.0x (configurable via constructor)
• Future: Could make configurable via application.yaml

Key decisions:

• Only multi-word phrases get expanded (single words have no benefit)
• User-specified slop always honored (backward compatible)
• Boost ensures exact matches always rank highest
• Slop of 3 balances recall (find variations) vs precision (avoid noise)

Key files: ProximityExpandingQueryParser.java, LuceneIndexService.java, ProximityExpandingQueryParserTest.java, AutomaticPhraseExpansionIntegrationTest.java

Alternative considered: Solr ComplexPhraseQueryParser - rejected due to Solr dependency, less control over scoring, and SpanQuery limitations with stopwords

Missing Features & Gaps

Critical Gaps (Should be added to Tier 1)

A. Index Backup & Restore

Status: Not implemented Priority: High Impact: Critical for production use

Currently no way to backup or restore the index. Users risk data loss on corruption or system failure.

Proposed tools:

• backupIndex - Create snapshot of index to specified location
• restoreIndex - Restore index from backup
• listBackups - List available backups with timestamps

Considerations:

• Index must be locked during backup
• Incremental backups vs full snapshots
• Backup verification/integrity checks
• Storage location configuration

Key files: New tools in LuceneSearchTools.java, new backup service

B. Duplicate Document Detection

Status: Not implemented Priority: Medium-High Impact: High

Index has content_hash field but no tools to find duplicate documents.

Proposed tool: findDuplicates

{
  "method": "content_hash",  // or "fuzzy_content", "title_similarity"
  "threshold": 0.95,
  "groupBy": "content_hash"
}

Returns groups of duplicate documents with suggestions for which to keep/remove.

Use cases:

• Cleanup after indexing multiple document sources
• Detect near-duplicates (different versions of same document)
• Index optimization (remove redundant documents)

Key files: New tool, new analysis method in LuceneIndexService.java

Tier 2 Additions (Medium Impact)

C. Batch/Bulk Operations

Status: Not implemented Priority: Medium Impact: Medium

No support for batch operations - each operation requires separate tool call.

Proposed features:

• batchSearch - Run multiple queries in one call, return combined results
• batchProfileQuery - Profile multiple queries for comparison
• batchGetDocuments - Retrieve multiple documents by file path

Benefits:

• Reduced MCP round-trips
• More efficient for comparative analysis
• Better for reporting/export scenarios

D. Export & Report Generation

Status: Not implemented Priority: Medium Impact: Medium

No way to export search results, profiling data, or statistics for external use.

Proposed tool: exportResults

{
  "source": "last_search",  // or "profile_results", "index_stats"
  "format": "csv",          // or "json", "markdown"
  "fields": ["file_path", "score", "language"],
  "destination": "/path/to/export.csv"
}

Use cases:

• Generate reports for stakeholders
• Export for external analysis (Excel, BI tools)
• Archive search results

E. Multi-Language Snowball Stemming (Per-Language Shadow Fields)

Status: ✅ DONE — implemented in SCHEMA_VERSION 3 Priority: Medium-High Impact: High (especially for German)

Implemented approach: Per-language lemma shadow fields using OpenNLP (replaced original Snowball stemming plan). All documents are indexed with both German and English lemma fields (content_lemma_de, content_lemma_en) regardless of detected language. At query time, weighted OR queries combine exact matching on content (boost 2.0) with lemmatized matching on language-specific fields (dynamic boost based on language distribution). Exact matches always rank highest. Highlighting stays on the unstemmed content field.

See PIPELINE.md for complete analyzer chain documentation, token examples, and query pipeline details.

Key implementation decisions:

• Dynamic boost weights derived from index language distribution: boost = 0.3 + 0.7 * (langCount / totalDocs)
• Language distribution cached and refreshed on NRT searcher refresh
• If explicit language eq "xx" filter present, only that language's lemma field is included at boost 1.0
• Highlighting unaffected (uses unstemmed content field)
• Documents found only via lemma fields get fallback passages (no bold markers)

E1b. OpenNLP Lemmatizer Token Cleanup

Status: Done (SCHEMA_VERSION 8) Effort: Low Impact: Medium — eliminates index noise, improves term observability tools

Problem: OpenNLPTokenizer retains punctuation as separate tokens, and German UD-GSD lemmatizer produces compound lemmas for contractions (im → in+der).

Solution: Added TypeTokenFilter (drops punctuation) and CompoundLemmaSplittingFilter (splits on +) to the lemmatizer chain.

See PIPELINE.md for complete analyzer chain details and examples.

E2. Irregular Verb Stemming (Extends Snowball Stemming)

Status: Superseded — OpenNLP lemmatization handles irregular verbs correctly Priority: N/A Impact: N/A (problem solved by lemmatizer) Depends on: Candidate E (Snowball stemming — replaced by OpenNLP lemmatization)

Problem: Snowball is an algorithmic suffix-stripper. It handles regular morphology well but fails on irregular forms that involve vowel changes (Ablaut), suppletion, or prefix patterns:

English gaps:

• ran → "ran" vs run/running → "run" (vowel change)
• went → "went" vs go/going → "go" (suppletion)
• saw → "saw" vs see/seen → "see" (vowel change)
• wrote → "wrote" vs write/written → "write"
• analysis → "analysi" vs analyses → "analys" (confirmed in P/R tests — different stems)

German gaps (more severe due to Ablaut + ge- prefix):

• ging → "ging" vs gehen/gegangen → different stems
• fuhr → "fuhr" vs fahren/gefahren → different stems
• sprach → "sprach" vs sprechen/gesprochen → different stems
• war → "war" vs sein/gewesen → completely unrelated forms (suppletion)
• lief → "lief" vs laufen/gelaufen → different stems

Current P/R state: P=0.975, R=1.000, F1=0.975 on the 45-doc test corpus. R=1.000 because the test corpus is small and queries are carefully chosen. In a real corpus, verb-based searches like "who ran the project" or "was entschieden wurde" would hit these gaps.

Approach A: Hunspell Dictionary Stemming

Use Lucene's built-in HunspellStemFilter with .dic/.aff dictionary files. Dictionary-based stemming correctly maps irregular forms to their lemma because it uses lookup tables rather than suffix rules.

Implementation: Additional shadow fields using HunspellStemFilter. See PIPELINE.md for analyzer chain details.

Pros:

• Handles all irregulars via dictionary lookup — no manual maintenance of verb lists
• Well-maintained dictionaries exist (LibreOffice/SCOWL communities)
• Lucene-native — HunspellStemFilter is in lucene-analysis-common (already a dependency)
• Covers not just irregular verbs but also irregular noun plurals, adjective inflections, etc.
• Would also improve Snowball's edge cases (e.g., analysis/analyses)

Cons:

• License problem for German: The de_DE Hunspell dictionary (igerman98) is GPL-2.0 OR GPL-3.0. Per the ASF 3rd Party License Policy, GPL is Category X — prohibited for bundling in Apache 2.0 projects. This applies to both code and data files.
• English dictionary (SCOWL) is MIT + BSD — Apache Category A, fully compatible.
• Slower than Snowball (dictionary lookups vs rule application)
• Dictionary files add ~5-10 MB per language
• Can produce multiple stems per token (ambiguity: "saw" → "see" AND "saw")
• Dictionary updates need tracking

License workaround options:

1. User-provided dictionaries: Ship without dictionaries, load from config path (hunspell-dictionaries-path in application.yaml). User downloads and places dictionaries themselves. GPL applies to the dictionary files, not to code that reads them.
2. English only: Bundle only the MIT/BSD English dictionary; German stays Snowball-only.
3. Wait for re-licensing: Monitor igerman98 — if it ever moves to LGPL or more permissive license.

If user-provided dictionaries approach:

# application.yaml
hunspell:
  enabled: false  # opt-in
  dictionaries-path: ~/.mcplucene/dictionaries/
  # User places de_DE.dic, de_DE.aff, en_US.dic, en_US.aff there

Approach B: StemmerOverrideFilter (Explicit Mapping)

Lucene's StemmerOverrideFilter runs before Snowball and provides a hard override map. Only irregular forms need entries; regular forms fall through to Snowball unchanged.

Implementation: Insert StemmerOverrideFilter into the existing StemmedUnicodeNormalizingAnalyzer chain, before SnowballFilter:

StandardTokenizer → LowerCaseFilter → ICUFoldingFilter → StemmerOverrideFilter(map) → SnowballFilter

Override map examples:

// English (~200 irregular verbs, ~5-6 forms each ≈ ~1000 entries)
"ran" → "run", "went" → "go", "gone" → "go",
"saw" → "see", "seen" → "see",
"wrote" → "write", "written" → "write",
"paid" → "pay", "analyses" → "analysis", ...

// German (~170 strong/mixed verbs, ~5-6 forms each ≈ ~1000 entries)
"ging" → "gehen", "gegangen" → "gehen",
"fuhr" → "fahren", "gefahren" → "fahren",
"sprach" → "sprechen", "gesprochen" → "sprechen",
"war" → "sein", "gewesen" → "sein",
"lief" → "laufen", "gelaufen" → "laufen", ...

Pros:

• No license issues — override maps are our own code, Apache 2.0
• Precise — no false conflations beyond what we explicitly define
• Composable — sits in front of existing Snowball chain, no new fields needed
• Tiny overhead (HashMap lookup per token)
• No new dependencies
• Easy to test — each override is a deterministic mapping

Cons:

• Manual maintenance of override maps (~2000 entries total for DE+EN)
• Incomplete coverage — only catches forms we enumerate
• Doesn't scale to new languages without per-language work
• The "long tail" of irregulars is large: English ~200 verbs × ~5 forms, German ~170 verbs × ~6 forms
• Doesn't cover irregular noun plurals or adjective forms (only verbs, unless we add those too)
• Override map targets must match Snowball's output stem for regular words (e.g., "went" → "go", but Snowball stems "go" to "go", "goes" to "goe" — we'd need "went" → "go" AND update "goes" → "go" to normalize Snowball's irregular output)

Key consideration: The override target must be the Snowball stem of the base form, not the dictionary lemma. E.g., if Snowball stems "house" → "hous", then override "houses" → "hous" (not "house"). Since StemmerOverrideFilter runs BEFORE SnowballFilter, overridden tokens skip Snowball entirely — so the target must be the final desired stem. This means the override map is Snowball-version-dependent.

Approach C: OpenNLP Lemmatizer (Lucene-integrated, Apache 2.0) — IMPLEMENTED

Lucene's OpenNLPLemmatizerFilter provides true lemmatization via trained models. This approach was chosen and implemented.

Implementation: Shadow fields content_lemma_de / content_lemma_en using OpenNLP pipeline with sentence detection, POS tagging, and lemmatization. See PIPELINE.md for complete analyzer chain documentation and token examples.

Critical difference from Snowball: Requires complete analyzer chain with OpenNLPTokenizer (sentence detection + tokenization) + POS tagging before lemmatization.

Two modes available:

1. Dictionary-only: DictionaryLemmatizer — HashMap lookup from word[tab]postag[tab]lemma file. Fast (O(1) per token), but requires POS tags and a comprehensive dictionary file.
2. MaxEnt model: LemmatizerME — statistical model (Maximum Entropy). Handles unseen words. Can be combined with dictionary (dictionary tried first, model for OOV fallback).

Available models (all Apache 2.0):

Memory and performance analysis:

Startup impact:

• Each language requires 4 models loaded into memory: sentence detector, tokenizer, POS tagger, lemmatizer
• Estimated heap per language: ~20-50 MB for small/medium models, ~150+ MB for DE Wikipedia model
• Models are loaded once at startup and shared across threads (MaxentModel is thread-safe)
• Current server startup: ~2 seconds. Model loading could add 1-3 seconds per language
• Total memory impact: 40-100 MB additional heap for both languages with medium-sized models. This is significant for a server that currently runs lean (50-100 MB heap)

Index-time impact (critical path):

• POS tagging is the bottleneck: MaxEnt model inference per token, not just a lookup
• Estimated throughput: ~10,000-50,000 tokens/second (vs ~500,000+ for Snowball)
• For a 10-page PDF (~5,000 tokens): Snowball ~10ms, OpenNLP ~100-500ms
• Full reindex of 10,000 documents: could add 15-60 minutes to crawl time
• POS tagging accuracy affects lemmatization: wrong POS tag → wrong lemma

Query-time impact:

• Query strings are short (typically 1-5 tokens) — OpenNLP pipeline overhead is negligible per query
• But the analyzer must be invoked for each stemmed field query (same as current Snowball approach)
• Estimated: ~1-5ms additional per query (acceptable)

Index size impact:

• Lemmatized tokens are typically shorter than or equal to surface forms (same as stemming)
• No significant additional index size vs Snowball shadow fields

Pros:

• All Apache 2.0 — library, Lucene integration, EN models, DE models (DE-Lemma)
• True lemmatization: handles ALL irregulars (verbs, nouns, adjectives) correctly
• Handles unseen words via MaxEnt model (not limited to dictionary entries)
• German models trained on 36.1M sentences (Wikipedia) — extensive vocabulary coverage
• Lucene-native TokenFilter — same integration pattern as Snowball
• Dictionary + model hybrid: fast lookup for known words, statistical fallback for unknown

Cons:

• Heavy pipeline: requires sentence detector + tokenizer + POS tagger + lemmatizer (4 models per language vs 0 for Snowball)
• Memory overhead: ~40-100 MB additional heap for both languages (doubles current footprint)
• Index-time performance: 10-50x slower than Snowball per token due to POS tagging
• Startup time: +1-3 seconds for model loading
• New dependency: lucene-analysis-opennlp module + model files bundled or downloaded
• Model files: must be bundled in JAR or downloaded on first run (~20-60 MB total for medium models)
• POS accuracy dependency: lemmatization quality depends on POS tagging accuracy. Wrong POS → wrong lemma (e.g., "saw" tagged as NN → "saw", tagged as VBD → "see")
• Complexity: separate complete analyzer (can't just add a filter to existing chain), language detection must route to correct analyzer
• Contradicts "fast, simple server" principle more than other approaches

Comparison

Recommendation: Start with Approach B (StemmerOverrideFilter) for both languages — no license issues, no dependencies, minimal performance impact, precise control. Cover the 50 most common irregular verbs per language first (90% of practical recall gap). The AI client handles the remaining long tail through query expansion.

If broader coverage is needed later: Approach C (OpenNLP) is the strongest fully-Apache-2.0 option for complete irregular form handling. The memory and performance costs are significant but may be acceptable if the server handles large corpora where irregular verb recall matters. Consider making it opt-in via configuration (lemmatizer.enabled: true, lemmatizer.engine: opennlp) with model files loaded on demand.

English-only Hunspell (Approach A) remains viable for English specifically (MIT/BSD license), with user-provided dictionaries for German as a power-user option.

Client-side complementary approach: Regardless of server-side choice, the AI client can always expand irregular forms via OR queries ("ran OR run OR running"). This is already the approach for synonyms and works well for the irregular verbs that server-side stemming doesn't cover.

OpenNLP as a Platform Investment: NLP-vs-LLM Boundary Analysis

Note: This analysis was written for server deployment scenarios. For the desktop/Claude Desktop use case, most OpenNLP features beyond basic lemmatization are rejected due to resource constraints (see "Explicitly Rejected" section). The architectural principle remains valid, but the desktop context shifts the boundary heavily toward client-side LLM processing.

If the OpenNLP pipeline were committed to in a server deployment (not the current desktop use case), it becomes a platform investment — not just "better stemming" but a foundation for multiple features. The key insight: paying the startup/memory cost once unlocks capabilities beyond lemmatization.

The Principle: "NLP enriches the index, LLM enriches the query"

NLP processing at index time adds structure to unstructured data — this structure lives permanently in the index and benefits every future query. LLM processing at query time adds intelligence from context — understanding user intent, expanding queries semantically, iterating on results. These are complementary, not competing.

The boundary criterion: If a capability produces deterministic, cacheable, document-level structure, it belongs in the NLP layer. If it requires contextual reasoning, user intent, or cross-document synthesis, it belongs in the LLM layer.

Feature 2: Sentence Detection for Document Chunking

OpenNLP's SentenceDetectorME identifies sentence boundaries — a critical primitive for document chunking (Tier 1 candidate #3).

Why it matters:

• Current chunking would split at arbitrary character boundaries → sentences cut mid-word
• Sentence-aware chunking produces semantically coherent passages
• Better chunks → better search passages → more useful results for the AI client

Why NLP, not LLM:

• Sentence detection is a well-solved NLP problem (>99% accuracy for EN/DE)
• Deterministic, fast, runs once at index time
• LLM-based sentence splitting would be absurdly expensive for every document
• The model is already loaded for the lemmatization pipeline (zero marginal cost)

Synergy with existing chunking candidate:

• Chunking at sentence boundaries with overlap → each chunk is 5-10 sentences
• Paragraphs detected via whitespace + sentence boundaries
• Chunk metadata includes sentence count, position in document

Impact assessment:

• Value: High (prerequisite for quality chunking)
• Effort: Near-zero if OpenNLP pipeline exists (sentence model already loaded)
• Memory: 0 additional (already loaded for lemmatizer tokenizer)
• Index time: Included in tokenizer pass

Feature 3: POS-Based Field Indexing

With POS tags available, index only specific parts of speech into specialized fields.

Examples:

• content_nouns — only nouns, for concept-level search
• content_verbs — only verbs, for action-level search

Assessment: Marginal value

• The AI client can already focus searches via query formulation
• BM25 naturally weights content-bearing terms (nouns) higher than function words
• Additional fields increase index size without clear user benefit
• The search interface has no natural way to express "search only nouns"

Verdict: Not recommended. The complexity-to-benefit ratio is too low. If specific use cases emerge (e.g., domain-specific terminology extraction), reconsider.

Feature 4: Noun Phrase Extraction

Extract multi-word noun phrases (e.g., "Arbeitsvertrag", "supply chain management") as atomic units.

Potential value:

• Index "supply chain management" as a single term → exact phrase matching without proximity operators
• Extract compound nouns that are written as separate words in English but would be single words in German

Assessment: Moderate value, but overlaps with LLM capabilities

• The AI client already handles phrase queries ("supply chain management")
• OpenNLP chunker models have moderate accuracy for complex phrases
• German compound nouns are already single tokens (no extraction needed)
• English multi-word terms benefit, but the AI can formulate phrase queries

Verdict: Interesting but not high-priority. Could add value for automated keyword extraction / tag generation at index time.

NLP-vs-LLM Boundary Summary

…(truncated)