Architecture

Architecture Deep Dive

System Overview

┌─────────────────────────────────────────────────────┐
│           Claude Desktop / MCP Client               │
└────────────────┬────────────────────────────────────┘
                 │ STDIO Transport (JSON-RPC)
                 ▼
┌─────────────────────────────────────────────────────┐
│           MCP Java SDK (STDIO Transport)            │
│  ┌──────────────────────────────────────────────┐   │
│  │        LuceneSearchTools (MCP Tools)         │   │
│  └──────────────────────────────────────────────┘   │
└─────────┬───────────────────────────┬───────────────┘
          │                           │
          ▼                           ▼
┌──────────────────────┐    ┌──────────────────────┐
│  LuceneIndexService  │    │ DocumentCrawler      │
│  - Search & Index    │    │ Service              │
│  - NRT Manager       │    │ - File Discovery     │
│  - Admin Operations  │    │ - Content Extraction │
└──────────┬───────────┘    └──────────┬───────────┘
           │                           │
           ▼                           ▼
┌─────────────────────────────────────────────────────┐
│              Apache Lucene 10.3 + Apache Tika       │
└─────────────────────────────────────────────────────┘

Design Decisions

Why Plain Java (no Spring)?

• Fast startup (~1 second) - critical for MCP subprocess
• Smaller JAR (~45MB)
• Direct control over lifecycle
• Trade-off: Manual dependency wiring in LuceneserverApplication.java

Why STDIO Transport?

• Required for Claude Desktop integration
• No network security concerns
• Consequence: Console logging MUST be disabled in production (deployed profile)

Why Multi-Analyzer Pipeline?

The server uses multiple analyzer chains optimized for different search patterns. Each document is indexed with several shadow fields, each using a different analyzer:

• UnicodeNormalizingAnalyzer: Primary content field with NFKC normalization, diacritic folding, ligature expansion
• ReverseUnicodeNormalizingAnalyzer: Reversed tokens for efficient leading wildcard queries (*vertrag)
• OpenNLPLemmatizingAnalyzer: Dictionary-based lemmatization for German and English (handles irregular forms)
• GermanTransliteratingAnalyzer: Maps ASCII digraphs to umlauts (Mueller→Müller)

See PIPELINE.md for complete analyzer chain documentation and query pipeline details.

Processing Patterns

Batch Processing

Directory Walkers (N threads) ──> LinkedBlockingQueue ──> Batch Processor (1 thread)

• Reduces Lucene commit overhead (commits are expensive)
• Configurable via batch-size and batch-timeout-ms

NRT (Near Real-Time) Optimization

• Normal: 100ms refresh interval
• Bulk indexing (>=1000 files): Auto-switches to 5000ms
• Prevents CPU thrashing during large crawls

Configuration Priority

Environment Variable > ~/.mcplucene/config.yaml > application.yaml

When LUCENE_CRAWLER_DIRECTORIES env var is set, MCP config tools return errors.

Admin Operations Pattern

• Long-running ops (optimize, purge) run async in single-threaded executor
• Tools return immediately with operationId
• Clients poll with getIndexAdminStatus()
• Only one admin operation can run at a time

Crawler Architecture

The document crawler uses a multi-layered architecture:

1. DocumentCrawlerService - Main orchestrator

   • Manages crawl lifecycle (start, pause, resume, stop)
   • Coordinates parallel directory processing
   • Handles batch queuing and processing
   • Manages NRT optimization

2. FileContentExtractor - Apache Tika integration

   • Extracts text content from documents
   • Detects document language
   • Extracts metadata (author, title, dates, etc.)

3. DocumentIndexer - Lucene document builder

   • Creates standardized Lucene documents
   • Handles document updates via content hash
   • Manages field schema

4. DirectoryWatcherService - File system monitoring

   • Uses Java WatchService for efficient monitoring
   • Handles file create, modify, delete events
   • Supports recursive directory watching

5. CrawlExecutorService - Thread pool management

   • Configurable worker threads
   • Bounded queue with backpressure handling

6. CrawlStatisticsTracker - Progress tracking

   • Thread-safe statistics collection
   • Automatic progress notifications
   • Per-directory breakdown

7. IndexReconciliationService - Incremental indexing

   • Compares index snapshot with filesystem in memory
   • Computes ADD / UPDATE / DELETE / SKIP sets
   • Applies bulk orphan deletions before new content is indexed
   • Designed to be fast: no content extraction during the reconciliation phase

MCP Response Token Budget

Every byte returned by an MCP tool is a token consumed by the AI client. This is a first-class design constraint — treat response size the same way you would treat latency or memory.

Rules for MCP tool responses:

• Never return full document content in search results. Return focused passages instead.
• Passages are truncated to max-passage-char-length (default: 200) using a highlight-centred window that trims irrelevant leading/trailing text while preserving <em> tags and surrounding context.
• Default max-passages is 3 per document — enough for relevance judgement, not exhaustive coverage. Users who need more can retrieve the full document via getDocument.
• When adding new fields to tool responses, ask: "Does the AI client actually need this in every response, or only on demand?" Prefer separate detail-fetching tools over bloating the default response.
• When designing new MCP tools, estimate the worst-case response size: (fields per result) × (avg field size) × (max results) × (max passages). Keep it under ~10 KB per tool call as a guideline.
• Prefer structured, machine-readable output (short field values, numeric scores) over verbose human-readable text.

Example: The passage system was redesigned to reduce worst-case per-search token load from ~25,000 chars (5 passages × 500 chars × 10 results) to ~6,000 chars (3 passages × 200 chars × 10 results) — a 76% reduction with better precision because passages are now individually extracted and windowed around highlights.

Limitations (Design Constraints)

Future Enhancement Ideas

See the /improvements skill for the full prioritized roadmap, trade-off analysis, and rejected ideas with reasoning.

Converted and distributed by TomeVault — claim your Tome and manage your conversions.