메뉴
Roadmap and analysis of potential improvements, design trade-offs, and feature priorities. Use when planning new features, evaluating what to build next, or understanding why certain approaches were chosen or rejected.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Detailed architecture documentation including design decisions, processing patterns, and system internals. Use when discussing architecture, understanding why something was built a certain way, or planning significant changes.
Guides for common development tasks like adding facet fields, file format support, STDIO debugging, and performance tuning. Use when working on specific development recipes.
Step-by-step guide for adding a new MCP tool endpoint. Use when creating a new tool that Claude Desktop or other MCP clients can call.
| name | improvements |
| description | Roadmap and analysis of potential improvements, design trade-offs, and feature priorities. Use when planning new features, evaluating what to build next, or understanding why certain approaches were chosen or rejected. |
| allowed-tools | Read, Glob, Grep |
The fundamental architecture decision is that semantic understanding lives in the MCP client (the AI), not in the server. The server is a fast, precise, lexical retrieval engine. The AI compensates for what the server doesn't do.
This means:
Any proposed improvement must be evaluated against this principle. If the AI client can do it, the server shouldn't duplicate it.
score, matchedTerms, termCoverage, position per passage is optimized for LLM consumptioncontent_reversed field for efficient *vertrag-style queries (German compound words)These improve the AI client's ability to use the server effectively without duplicating semantic logic.
Status: Done Effort: Medium Impact: High
Implemented filters[] array with operators (eq, in, not, not_in, range), DrillSideways faceting for faceted fields, ISO-8601 date parsing, activeFilters with matchCount in response, dateFieldHints in getIndexStats, and backward compatibility with legacy filterField/filterValue. Also addresses item 6 (Date-Friendly Query Parameters) via the range operator with ISO-8601 support.
Status: Done Effort: Low-Medium Impact: High
Implemented two read-only MCP tools for index vocabulary exploration:
suggestTerms — Prefix-based term completion with doc frequency sorting. Auto-lowercases prefix for analyzed fields (using LOWERCASE_WILDCARD_FIELDS). Cross-segment aggregation via HashMap merge. Point fields rejected with helpful error message.
getTopTerms — Full term enumeration sorted by frequency. Warning for fields with >100K unique terms. Works with both analyzed and StringFields.
Both tools: empty results for nonexistent fields (graceful), validation rejects LONG_POINT_FIELDS, new IndexObservabilityIntegrationTest with 20 tests covering prefix matching, case handling, limits, empty index, cross-segment aggregation.
Status: Not started Effort: High Impact: High
Currently each file = one Lucene document. The highlighter reads only 10,000 chars (withMaxLength), so passages from the second half of long PDFs are never found.
Approach: Split documents into overlapping chunks (e.g., ~2000 chars with 200-char overlap) at paragraph boundaries. Each chunk is a Lucene document linked to its parent by file_path. Search returns chunk-level passages; the AI sees which section of the document is relevant.
Considerations:
getDocumentDetails would need to reconstruct the full document from chunksKey files: DocumentIndexer.java, LuceneIndexService.search(), DocumentCrawlerService
Status: ✅ Done Effort: Low Impact: Medium-High
Implemented sorting capability for search results by metadata fields.
What was implemented:
sortBy parameter: _score (default), modified_date, created_date, file_sizesortOrder parameter: asc or desc (defaults: desc for dates/size, desc for score)Examples:
// Most recently modified
{"query": "contract", "sortBy": "modified_date", "sortOrder": "desc"}
// Oldest documents
{"query": "*", "sortBy": "created_date", "sortOrder": "asc"}
// Smallest files
{"query": "summary", "sortBy": "file_size", "sortOrder": "asc"}
Key decisions:
SortedNumericSortField for numeric/date fields (DocValues)Key files: SearchRequest.java, LuceneIndexService.search(), README.md, query-syntax resource
Status: Not started Effort: Medium Impact: Medium
Lucene has MoreLikeThis built in. A findSimilar tool that takes a file_path and returns related documents would be useful for the AI ("find documents similar to this contract").
Key files: New tool in LuceneSearchTools.java, new method in LuceneIndexService.java
Status: Done (addressed by Structured Multi-Filter Support) Effort: Low Impact: Medium
Implemented as part of the filters[] array with range operator and ISO-8601 date parsing. The getIndexStats tool now also returns dateFieldHints with min/max dates.
Status: Not started Effort: Medium-High Impact: Medium (depends on user's document mix)
Tika supports Tesseract OCR. Scanned PDFs are a blind spot — the content extractor returns empty text. Even basic OCR would dramatically expand coverage for users with scanned documents.
Considerations:
ocr-enabled: true)Key files: FileContentExtractor.java, application.yaml
Status: Done Effort: Low Impact: Low-Medium
Added default support for 8 new file formats (all handled natively by Tika 3.2.3, no new dependencies):
.eml, .msg (emails).md, .rst (markup).html, .htm (web pages).rtf (rich text).epub (ebooks)Includes include-patterns in application.yaml + ApplicationConfig.java, test document generators, parameterized extraction tests, and README documentation.
Remaining: .csv (spreadsheets as text) could still be added.
Key files: application.yaml, ApplicationConfig.java, TestDocumentGenerator.java, FileContentExtractorTest.java, README.md
Status: ✅ Done Effort: Low Impact: Medium
Changed search result highlighting from HTML <em> tags to markdown **bold** syntax for proper rendering in Claude Desktop.
What was implemented:
IndividualPassageFormatter to use ** instead of <em> tagsextractMatchedTerms() in LuceneIndexService to parse markdown bold markersRationale:
**bold** syntax renders properly as visual emphasisKey files: IndividualPassageFormatter.java, LuceneIndexService.java, Passage.java, test files
Key decisions:
<b> tags to markdown **Status: ✅ Done Effort: High Impact: High
Implemented profileQuery MCP tool with multi-level analysis optimized for LLM/human readability.
What it provides:
Key decisions:
Explanation.toString() into structured DTOs instead of exposing raw format (version-independent)What it enables:
Files: 15 new DTOs in mcp/dto/, updates to LuceneSearchTools.java and LuceneIndexService.java
Known limitations (from Lucene's Explanation API):
*vertrag* in "Arbeitsvertrag"Future enhancements (see Tier 3 below):
Key files: ProfileQueryRequest.java, ProfileQueryResponse.java, QueryAnalysis.java, DocumentScoringExplanation.java, and 11 other DTOs
Status: ✅ Done Effort: Medium Impact: High
Implemented MCP Resources pattern to move verbose documentation out of tool descriptions, reducing initial context load by ~70%.
What was implemented:
lucene://docs/query-syntax - 350+ line Lucene query syntax guidelucene://docs/profiling-guide - 150+ line profiling analysis guideRead toolPattern established:
Benefits:
Key files: LuceneSearchTools.java (resource specifications and handlers), all DTO files (shortened descriptions)
Future applications:
Status: Not started Effort: Medium Impact: Medium-High
Enhance visual presentation of search results in Claude Desktop through rich markdown formatting and optional document previews.
Proposed enhancements:
Tier 1 (High Value, Low Cost) - Implement First:
Rich Markdown Formatting - Format search results with proper markdown structure:
Structured Result Cards - Clear visual layout:
### 📄 Document Title
**Path:** `/path/to/file.pdf`
**Modified:** 2024-01-15 | **Size:** 2.3 MB
**Preview:**
The **budget** for Q4 2024 was exceeded...
Better Snippets - Enhanced passage formatting:
Tier 2 (Medium Value, Medium Cost) - Optional: 4. File Type Icons - Embed tiny base64 icons (5-10 common types, ~50 KB total) 5. Search Statistics - Formatted facet counts, distribution charts as text
Tier 3 (High Value, High Cost) - Make Optional: 6. Document Thumbnails - MCP supports base64-encoded images in responses
thumbnails.enabled: false (default)MCP Support: MCP tool responses support multiple content types:
{"type": "text", "text": "..."} - Markdown-formatted text{"type": "image", "data": "base64...", "mimeType": "image/jpeg"} - ImagesRationale:
Key files: LuceneSearchTools.java, new ThumbnailGenerator.java (if implementing thumbnails), SearchResponse.java
Implementation priority: Start with Tier 1 (zero cost, immediate value), add thumbnails only if users request it.
Status: ✅ Done Effort: Low-Medium Impact: Medium-High
Implemented automatic expansion of exact phrase queries to include proximity matching, improving recall while maintaining precision through differential scoring.
What was implemented:
ProximityExpandingQueryParser extending Lucene's QueryParser"Domain Design" → ("Domain Design")^2.0 OR ("Domain Design"~3)Rationale:
Real-world example:
Query: "Domain Design"
Results by score:
1. "Domain Design" (exact) - Score: 0.6981 ⭐⭐⭐
2. "Domain-driven Design" - Score: 0.1360 ⭐⭐
3. "Domain Effective Design" - Score: 0.1360 ⭐⭐
4. "Domain Very Effective Design" - Score: 0.0864 ⭐
Configuration:
Key decisions:
Key files: ProximityExpandingQueryParser.java, LuceneIndexService.java, ProximityExpandingQueryParserTest.java, AutomaticPhraseExpansionIntegrationTest.java
Alternative considered: Solr ComplexPhraseQueryParser - rejected due to Solr dependency, less control over scoring, and SpanQuery limitations with stopwords
Status: Not implemented Priority: High Impact: Critical for production use
Currently no way to backup or restore the index. Users risk data loss on corruption or system failure.
Proposed tools:
backupIndex - Create snapshot of index to specified locationrestoreIndex - Restore index from backuplistBackups - List available backups with timestampsConsiderations:
Key files: New tools in LuceneSearchTools.java, new backup service
Status: Not implemented Priority: Medium-High Impact: High
Index has content_hash field but no tools to find duplicate documents.
Proposed tool: findDuplicates
{
"method": "content_hash", // or "fuzzy_content", "title_similarity"
"threshold": 0.95,
"groupBy": "content_hash"
}
Returns groups of duplicate documents with suggestions for which to keep/remove.
Use cases:
Key files: New tool, new analysis method in LuceneIndexService.java
Status: Not implemented Priority: Medium Impact: Medium
No support for batch operations - each operation requires separate tool call.
Proposed features:
batchSearch - Run multiple queries in one call, return combined resultsbatchProfileQuery - Profile multiple queries for comparisonbatchGetDocuments - Retrieve multiple documents by file pathBenefits:
Status: Not implemented Priority: Medium Impact: Medium
No way to export search results, profiling data, or statistics for external use.
Proposed tool: exportResults
{
"source": "last_search", // or "profile_results", "index_stats"
"format": "csv", // or "json", "markdown"
"fields": ["file_path", "score", "language"],
"destination": "/path/to/export.csv"
}
Use cases:
Status: ✅ DONE — implemented in SCHEMA_VERSION 3 Priority: Medium-High Impact: High (especially for German)
Implemented approach: Per-language lemma shadow fields using OpenNLP (replaced original Snowball stemming plan). All documents are indexed with both German and English lemma fields (content_lemma_de, content_lemma_en) regardless of detected language. At query time, weighted OR queries combine exact matching on content (boost 2.0) with lemmatized matching on language-specific fields (dynamic boost based on language distribution). Exact matches always rank highest. Highlighting stays on the unstemmed content field.
See PIPELINE.md for complete analyzer chain documentation, token examples, and query pipeline details.
Key implementation decisions:
boost = 0.3 + 0.7 * (langCount / totalDocs)language eq "xx" filter present, only that language's lemma field is included at boost 1.0content field)Status: Done (SCHEMA_VERSION 8) Effort: Low Impact: Medium — eliminates index noise, improves term observability tools
Problem: OpenNLPTokenizer retains punctuation as separate tokens, and German UD-GSD lemmatizer produces compound lemmas for contractions (im → in+der).
Solution: Added TypeTokenFilter (drops punctuation) and CompoundLemmaSplittingFilter (splits on +) to the lemmatizer chain.
See PIPELINE.md for complete analyzer chain details and examples.
Status: Superseded — OpenNLP lemmatization handles irregular verbs correctly Priority: N/A Impact: N/A (problem solved by lemmatizer) Depends on: Candidate E (Snowball stemming — replaced by OpenNLP lemmatization)
Problem: Snowball is an algorithmic suffix-stripper. It handles regular morphology well but fails on irregular forms that involve vowel changes (Ablaut), suppletion, or prefix patterns:
English gaps:
ran → "ran" vs run/running → "run" (vowel change)went → "went" vs go/going → "go" (suppletion)saw → "saw" vs see/seen → "see" (vowel change)wrote → "wrote" vs write/written → "write"analysis → "analysi" vs analyses → "analys" (confirmed in P/R tests — different stems)German gaps (more severe due to Ablaut + ge- prefix):
ging → "ging" vs gehen/gegangen → different stemsfuhr → "fuhr" vs fahren/gefahren → different stemssprach → "sprach" vs sprechen/gesprochen → different stemswar → "war" vs sein/gewesen → completely unrelated forms (suppletion)lief → "lief" vs laufen/gelaufen → different stemsCurrent P/R state: P=0.975, R=1.000, F1=0.975 on the 45-doc test corpus. R=1.000 because the test corpus is small and queries are carefully chosen. In a real corpus, verb-based searches like "who ran the project" or "was entschieden wurde" would hit these gaps.
Approach A: Hunspell Dictionary Stemming
Use Lucene's built-in HunspellStemFilter with .dic/.aff dictionary files. Dictionary-based stemming correctly maps irregular forms to their lemma because it uses lookup tables rather than suffix rules.
Implementation: Additional shadow fields using HunspellStemFilter. See PIPELINE.md for analyzer chain details.
Pros:
HunspellStemFilter is in lucene-analysis-common (already a dependency)analysis/analyses)Cons:
License workaround options:
hunspell-dictionaries-path in application.yaml). User downloads and places dictionaries themselves. GPL applies to the dictionary files, not to code that reads them.If user-provided dictionaries approach:
# application.yaml
hunspell:
enabled: false # opt-in
dictionaries-path: ~/.mcplucene/dictionaries/
# User places de_DE.dic, de_DE.aff, en_US.dic, en_US.aff there
Approach B: StemmerOverrideFilter (Explicit Mapping)
Lucene's StemmerOverrideFilter runs before Snowball and provides a hard override map. Only irregular forms need entries; regular forms fall through to Snowball unchanged.
Implementation: Insert StemmerOverrideFilter into the existing StemmedUnicodeNormalizingAnalyzer chain, before SnowballFilter:
StandardTokenizer → LowerCaseFilter → ICUFoldingFilter → StemmerOverrideFilter(map) → SnowballFilter
Override map examples:
// English (~200 irregular verbs, ~5-6 forms each ≈ ~1000 entries)
"ran" → "run", "went" → "go", "gone" → "go",
"saw" → "see", "seen" → "see",
"wrote" → "write", "written" → "write",
"paid" → "pay", "analyses" → "analysis", ...
// German (~170 strong/mixed verbs, ~5-6 forms each ≈ ~1000 entries)
"ging" → "gehen", "gegangen" → "gehen",
"fuhr" → "fahren", "gefahren" → "fahren",
"sprach" → "sprechen", "gesprochen" → "sprechen",
"war" → "sein", "gewesen" → "sein",
"lief" → "laufen", "gelaufen" → "laufen", ...
Pros:
Cons:
Key consideration: The override target must be the Snowball stem of the base form, not the dictionary lemma. E.g., if Snowball stems "house" → "hous", then override "houses" → "hous" (not "house"). Since StemmerOverrideFilter runs BEFORE SnowballFilter, overridden tokens skip Snowball entirely — so the target must be the final desired stem. This means the override map is Snowball-version-dependent.
Approach C: OpenNLP Lemmatizer (Lucene-integrated, Apache 2.0) — IMPLEMENTED
Lucene's OpenNLPLemmatizerFilter provides true lemmatization via trained models. This approach was chosen and implemented.
Implementation: Shadow fields content_lemma_de / content_lemma_en using OpenNLP pipeline with sentence detection, POS tagging, and lemmatization. See PIPELINE.md for complete analyzer chain documentation and token examples.
Critical difference from Snowball: Requires complete analyzer chain with OpenNLPTokenizer (sentence detection + tokenization) + POS tagging before lemmatization.
Two modes available:
DictionaryLemmatizer — HashMap lookup from word[tab]postag[tab]lemma file. Fast (O(1) per token), but requires POS tags and a comprehensive dictionary file.LemmatizerME — statistical model (Maximum Entropy). Handles unseen words. Can be combined with dictionary (dictionary tried first, model for OOV fallback).Available models (all Apache 2.0):
| Model | Source | Size | Notes |
|---|---|---|---|
| EN lemmatizer | OpenNLP models | ~1-5 MB (est.) | opennlp-en-ud-ewt-lemmas-1.3-2.5.4.bin |
| EN POS tagger | OpenNLP models | ~5-15 MB (est.) | opennlp-en-ud-ewt-pos-1.3-2.5.4.bin |
| EN sentence detector | OpenNLP models | ~1 MB (est.) | opennlp-en-ud-ewt-sentence-1.3-2.5.4.bin |
| EN tokenizer | OpenNLP models | ~1 MB (est.) | opennlp-en-ud-ewt-tokens-1.3-2.5.4.bin |
| DE lemmatizer (small) | DE-Lemma | 861 KB | UD-GSD based, Apache 2.0 |
| DE lemmatizer (large) | DE-Lemma | 14 MB | UD-HDT based, Apache 2.0 |
| DE lemmatizer (huge) | DE-Lemma | 131 MB | Wikipedia-trained (36.1M sentences), Apache 2.0 |
| DE POS tagger | OpenNLP models | ~5-15 MB (est.) | opennlp-de-ud-gsd-pos-1.3-2.5.4.bin |
| DE sentence detector | OpenNLP models | ~1 MB (est.) | opennlp-de-ud-gsd-sentence-1.3-2.5.4.bin |
| DE tokenizer | OpenNLP models | ~1 MB (est.) | opennlp-de-ud-gsd-tokens-1.3-2.5.4.bin |
Memory and performance analysis:
Startup impact:
Index-time impact (critical path):
Query-time impact:
Index size impact:
Pros:
TokenFilter — same integration pattern as SnowballCons:
lucene-analysis-opennlp module + model files bundled or downloadedComparison
| Aspect | Hunspell | StemmerOverrideFilter | OpenNLP Lemmatizer |
|---|---|---|---|
| Irregular verb coverage | Complete (dictionary) | ~90% of common verbs (manual) | Complete (trained model) |
| Irregular noun/adj coverage | Yes | Only if manually added | Yes |
| German license | GPL — blocked | Apache 2.0 (our code) | Apache 2.0 |
| English license | MIT/BSD — OK | Apache 2.0 (our code) | Apache 2.0 |
| Maintenance burden | Dictionary updates (community) | Manual map curation | Model updates (community) |
| Performance (index time) | Medium (dict lookup) | Minimal (HashMap) | Slow (10-50x vs Snowball) |
| Performance (query time) | Fast | Fastest | Fast (short queries) |
| Memory overhead | ~5-10 MB/lang (dict files) | None | ~20-50 MB/lang (models) |
| Startup impact | Minimal | None | +1-3 seconds |
| New dependencies | Dictionary files only | None | lucene-analysis-opennlp + 4 models/lang |
| False conflations | Possible (ambiguous words) | Only what we define | POS-dependent (mostly correct) |
| Complexity | Low (single filter) | Low (single filter) | High (full NLP pipeline) |
| Effort | Medium | Medium | High |
Recommendation: Start with Approach B (StemmerOverrideFilter) for both languages — no license issues, no dependencies, minimal performance impact, precise control. Cover the ~50 most common irregular verbs per language first (~90% of practical recall gap). The AI client handles the remaining long tail through query expansion.
If broader coverage is needed later: Approach C (OpenNLP) is the strongest fully-Apache-2.0 option for complete irregular form handling. The memory and performance costs are significant but may be acceptable if the server handles large corpora where irregular verb recall matters. Consider making it opt-in via configuration (lemmatizer.enabled: true, lemmatizer.engine: opennlp) with model files loaded on demand.
English-only Hunspell (Approach A) remains viable for English specifically (MIT/BSD license), with user-provided dictionaries for German as a power-user option.
Client-side complementary approach: Regardless of server-side choice, the AI client can always expand irregular forms via OR queries ("ran OR run OR running"). This is already the approach for synonyms and works well for the irregular verbs that server-side stemming doesn't cover.
Note: This analysis was written for server deployment scenarios. For the desktop/Claude Desktop use case, most OpenNLP features beyond basic lemmatization are rejected due to resource constraints (see "Explicitly Rejected" section). The architectural principle remains valid, but the desktop context shifts the boundary heavily toward client-side LLM processing.
If the OpenNLP pipeline were committed to in a server deployment (not the current desktop use case), it becomes a platform investment — not just "better stemming" but a foundation for multiple features. The key insight: paying the startup/memory cost once unlocks capabilities beyond lemmatization.
The Principle: "NLP enriches the index, LLM enriches the query"
NLP processing at index time adds structure to unstructured data — this structure lives permanently in the index and benefits every future query. LLM processing at query time adds intelligence from context — understanding user intent, expanding queries semantically, iterating on results. These are complementary, not competing.
The boundary criterion: If a capability produces deterministic, cacheable, document-level structure, it belongs in the NLP layer. If it requires contextual reasoning, user intent, or cross-document synthesis, it belongs in the LLM layer.
Feature 2: Sentence Detection for Document Chunking
OpenNLP's SentenceDetectorME identifies sentence boundaries — a critical primitive for document chunking (Tier 1 candidate #3).
Why it matters:
Why NLP, not LLM:
Synergy with existing chunking candidate:
Impact assessment:
Feature 3: POS-Based Field Indexing
With POS tags available, index only specific parts of speech into specialized fields.
Examples:
content_nouns — only nouns, for concept-level searchcontent_verbs — only verbs, for action-level searchAssessment: Marginal value
Verdict: Not recommended. The complexity-to-benefit ratio is too low. If specific use cases emerge (e.g., domain-specific terminology extraction), reconsider.
Feature 4: Noun Phrase Extraction
Extract multi-word noun phrases (e.g., "Arbeitsvertrag", "supply chain management") as atomic units.
Potential value:
Assessment: Moderate value, but overlaps with LLM capabilities
Verdict: Interesting but not high-priority. Could add value for automated keyword extraction / tag generation at index time.
NLP-vs-LLM Boundary Summary
| Capability | Server-Side (NLP) | Client-Side (LLM) | Winner |
|---|---|---|---|
| Lemmatization | Index-time, deterministic, all docs once | Per-query OR expansion ("ran OR run") | NLP — scales better |
| NER | Index-time extraction → facets | Read each doc, extract on demand | NLP — pre-extraction essential |
| Sentence detection | Index-time chunking boundaries | Not applicable | NLP — no alternative |
| POS-based fields | Separate noun/verb indexes | Query formulation targets concepts naturally | LLM — existing behavior sufficient |
| Noun phrase extraction | Index compound terms as units | Phrase queries, proximity operators | Draw — LLM already handles well |
| Synonym expansion | Static synonym files (brittle) | Context-aware OR queries | LLM — superior quality |
| Query refinement | Server can't understand intent | Iterative search-read-refine cycle | LLM — requires reasoning |
| Cross-document synthesis | Can't do at all | "Find documents about topic X and summarize" | LLM — inherently cross-doc |
| Language detection | Tika/OpenNLP at index time | Could guess from query text | NLP — per-document, deterministic |
| Relevance ranking | BM25 + boost weights | AI reads results, re-ranks by understanding | Both — BM25 for initial, LLM for refinement |
Key takeaway: NLP and LLM are complementary layers, not alternatives. NLP adds permanent structure to the index that every query benefits from. The LLM adds per-query intelligence that no static analysis can match. The highest-value NLP features are those that create new searchable dimensions (NER → entity facets) rather than those that duplicate LLM capabilities (synonym expansion, query refinement).
OpenNLP Platform Decision Framework
If committing to OpenNLP, the cost-benefit changes:
| Scenario | Lemmatization only | Lemma + NER | Lemma + NER + Chunking |
|---|---|---|---|
| Models loaded | 8 (4/lang) | 14-20 | 14-20 (same) |
| Memory overhead | ~40-100 MB | ~70-190 MB | ~70-190 MB |
| Startup time | +1-3 sec | +2-5 sec | +2-5 sec |
| Index-time overhead | 10-50x per token | +50-200ms/doc | Negligible additional |
| New searchable dimensions | 0 (recall improvement) | 3-6 entity facets | Better passages |
| Value multiplier | 1x | 3-5x | 5-7x |
The marginal cost of adding NER and sentence-aware chunking on top of an existing OpenNLP pipeline is low relative to the marginal value. This is why OpenNLP should be evaluated as a platform, not just a lemmatizer.
Recommendation: If Approach C (OpenNLP Lemmatizer) is ever pursued, plan for NER and sentence-aware chunking from the start. Design the pipeline, model loading, and configuration to support all three from day one, even if NER and chunking are initially disabled. The incremental cost of "ready for NER" is near zero; retrofitting later is much harder.
Status: Not implemented Priority: Medium Impact: Medium
Builds on #2 (Index Observability). Help users discover terms as they type.
Proposed tool: suggestQuery
{
"partial": "cont",
"field": "content",
"limit": 10
}
// Returns: ["contract", "content", "contractor", "contribute", ...]
Integration: Works well with suggestTerms (item #2) but optimized for prefix matching.
Key files: New tool, uses Lucene's TermsEnum.seekCeil()
Status: Not implemented Priority: Low-Medium Impact: Medium
Currently getIndexStats is point-in-time. No historical tracking.
Proposed enhancement:
getIndexTrends returns growth charts, query performance trendsUse cases:
Status: Not implemented Priority: Low Impact: Low-Medium
No support for tracking document versions or history.
Proposed approach:
content_hash or custom IDUse cases:
Challenges:
Status: Not implemented Priority: Low Impact: Low
Currently all results returned at once. Large result sets could benefit from streaming.
Challenge: MCP STDIO doesn't support streaming well. Would need chunked response pattern.
Alternative: Use pagination effectively (already implemented).
Status: Not explicitly supported/documented Priority: Low Impact: Low
Lucene supports regex queries but not mentioned in documentation.
Action needed:
Status: Not implemented Priority: Low Impact: Low
Currently uses Tika's default metadata extraction. No way to add custom fields.
Proposed approach:
Use cases:
PROJECT-123-contract.pdf)Complexity: High - requires plugin architecture, security sandboxing
Status: Not started Effort: Medium Impact: High
Currently profileQuery only explains documents that did match. Can't debug why an expected document didn't appear in results.
Proposed tool: explainNonMatch or extend profileQuery with explainNonMatch parameter:
{
"query": "contract signed",
"filters": [...],
"explainNonMatch": "/path/to/expected/doc.pdf"
}
Returns:
{
"documentPath": "/path/to/expected/doc.pdf",
"matched": false,
"reasons": [
{
"type": "filter_excluded",
"filter": {"field": "language", "value": "en"},
"actualValue": "de",
"explanation": "Document has language 'de', but filter requires 'en'"
},
{
"type": "missing_term",
"term": "signed",
"explanation": "Term 'signed' does not appear in document content"
}
],
"whatIfAnalysis": {
"withoutFilters": {"wouldMatch": true, "score": 2.3},
"withAllTermsOptional": {"wouldMatch": true, "score": 1.5}
}
}
Implementation: Retrieve document by file path, test each query component individually, test filters one-by-one, run "what-if" scenarios.
Benefit: Massive debugging improvement for complex boolean queries with filters.
Status: Not started Effort: Medium-High Impact: Medium
Hard to understand differences between similar queries or why results changed.
Proposed tool: compareQueries
{
"queryA": {"query": "*vertrag", "filters": [...]},
"queryB": {"query": "vertrag*", "filters": [...]}
}
Returns:
{
"differences": {
"totalHits": {"queryA": 450, "queryB": 380},
"uniqueToA": 70,
"uniqueToB": 0
},
"queryStructure": {
"queryA": "WildcardQuery(content_reversed:gartrev*)",
"queryB": "WildcardQuery(content:vertrag*)",
"difference": "Query A searches reversed field for leading wildcard"
},
"topDocumentChanges": [
{
"document": "/path/to/doc.pdf",
"rankInA": 5,
"rankInB": 1,
"explanation": "Ranks higher in B due to higher TF"
}
]
}
Benefit: A/B testing queries, understanding query behavior differences.
Status: Not started Effort: Medium Impact: Medium
profileQuery explains document-level BM25 scoring, but not why specific passages were selected or scored.
Current gap: UnifiedHighlighter computes passage scores internally but doesn't expose explanation. Can't answer "Why is passage 2 scored higher than passage 1?"
Proposed solution: Instrument IndividualPassageFormatter or create custom highlighter to capture:
Benefit: Complete scoring transparency from query → document → passage.
Status: Not started (basic recommendations implemented) Effort: Low-Medium Impact: Medium
Current profileQuery provides basic recommendations. Could be enhanced with:
Example output:
{
"recommendations": [
{
"type": "remove_common_term",
"severity": "high",
"issue": "Term 'the' appears in 95% of documents",
"suggestion": "Remove 'the' from query",
"estimatedImprovement": "50% faster execution"
}
]
}
Status: Not started Effort: Low Impact: Medium
Lucene query syntax is complex; users struggle with escaping, boolean logic, wildcard placement.
Proposed tool: buildQuery - constructs valid Lucene queries from structured input:
{
"operation": "AND",
"clauses": [
{"type": "phrase", "text": "signed contract", "proximity": 5},
{"type": "wildcard", "text": "vertrag", "position": "contains"},
{"type": "fuzzy", "text": "agreement", "maxEdits": 2}
]
}
Returns: {"generatedQuery": "\"signed contract\"~5 AND *vertrag* AND agreement~2", ...}
Benefit: Lower barrier to entry, fewer syntax errors.
Status: Not started Priority: Low Impact: Medium
Named queries with optional MCP notifications when new documents match. Turns the server from reactive search into a proactive knowledge assistant.
Challenges:
Status: Not started Priority: Low Impact: Low-Medium
Track queries, result counts, and opened documents. The AI could use this data to improve search strategies over time.
Challenges:
Status: Not started Priority: Low Impact: Low
A fields parameter to select which fields appear in results. Reduces response size for narrow use cases.
Current workaround: Response size is already reasonable; passages are the bulk of data, already controllable via pageSize.
Decision: Made filter impact, document scoring, and facet cost analysis opt-in (default: false)
Trade-offs:
Rationale: Profiling is a debugging tool, not a production feature. Most users want quick feedback; power users can enable deep analysis.
Decision: Parse Lucene's Explanation tree into structured DTOs instead of exposing toString()
Trade-offs:
Rationale: Explanation format has changed across Lucene versions. Parsing gives us control over output format and future-proofs the API.
Decision: Measure filter impact by running queries incrementally (no filter → filter 1 → filter 1+2 → ...)
Trade-offs:
Alternative considered: Lucene's cost() API - too inaccurate for filters
Rationale: Accuracy is critical for debugging; performance cost is acceptable since it's opt-in.
Decision: Categorize term rarity ("rare", "common", "very common") in addition to raw document frequency
Trade-offs:
Rationale: Raw numbers like "docFreq=4500, totalDocs=10000" require mental math. Categories are immediately actionable.
Decision: NLP enriches the index (structure from unstructured data), LLM enriches the query (intelligence from context). Features are evaluated against this boundary.
Boundary criterion: If a capability produces deterministic, cacheable, document-level structure → NLP layer. If it requires contextual reasoning, user intent, or cross-document synthesis → LLM layer.
High-value NLP features (create new searchable dimensions):
LLM-superior features (require reasoning):
Rationale: The highest-value server-side NLP features are those that create new searchable dimensions (like NER creating entity facets) — dimensions that don't exist without NLP processing. Features that merely duplicate what the LLM already does well (synonym expansion, query refinement) should stay client-side. See Candidate E2 for full analysis.
These are Lucene design limitations, not implementation choices:
Automaton internals are opaque: Wildcard/regex/fuzzy queries compile to finite automata (DFA/NFA). Once compiled, semantic meaning is lost. Cannot trace which specific substring matched in *vertrag* → "Arbeitsvertrag" vs "Mietvertrag". The automaton just has state transitions, no semantic labels.
Cannot explain non-matches: Explanation API requires a document ID from search results. Cannot explain why a document didn't match without manually testing query components.
No passage-level explanation: UnifiedHighlighter scores passages internally but doesn't expose explanation tree. Passage.score values are shown but not explained.
Performance overhead: Explanation is expensive (10-100x slower than just scoring). Uses reflection, creates deep object trees. Not suitable for all results or real-time use.
Explains rewritten queries: Shows the query after rewrites (e.g., content_reversed:gartrev* not *vertrag). Requires additional tracking to correlate with user's original query.
No filter selectivity metrics: Doesn't separately quantify filter contribution. Must run additional queries to measure filter impact.
Cross-document comparison: Each explanation is independent. Requires custom logic to compare "why doc A ranked higher than doc B".
Workarounds implemented:
queryAnalysisFuture work: Items 10-12 above address some of these gaps.
Decision: Rejected for desktop/Claude Desktop use case Reasoning:
The proposal: Extract entities (persons, organizations, locations) during indexing using OpenNLP NER models, store in faceted fields for searchable entity dimensions.
Why rejected:
Desktop hardware constraints - This MCP tool runs on consumer desktops, not servers:
Claude already does excellent NER - The AI client handles entity recognition on search results:
Division of labor - Fast search retrieval (server) + intelligent understanding (Claude)
Battery and performance - Laptop users care about:
When this would make sense:
Alternative approach: Let Claude do entity extraction on the top N search results (already happens naturally). The server focuses on fast, accurate text retrieval.
Related: The broader OpenNLP platform analysis (lemmatization, sentence detection, POS tagging) also needs reevaluation for desktop constraints. Snowball stemming (Candidate E - completed) provides good recall improvement without the OpenNLP overhead.
Decision: Rejected Reasoning: Lucene doesn't expose internal query execution timing at component level. The cost() API provides estimates, but actual execution time isn't broken down by query clause.
Alternative: Use overall execution time + cost estimates + incremental queries (implemented in profileQuery).
Decision: Rejected Reasoning:
Alternative: Parse Explanation into semantic DTOs (implemented).
search ToolDecision: Rejected Reasoning:
Alternative: Separate profileQuery tool (implemented).
Decision: Rejected Reasoning:
Alternative: Show original query + rewritten query; accept automaton internals are opaque.
Decision: Rejected for now Reasoning: Contradicts the core "client-side intelligence" architecture.
The AI client already handles semantic understanding. Adding embeddings would:
The gap vector search fills: finding documents where the concept matches but zero terms overlap, and the AI can't predict the document's vocabulary. Example: searching for "reducing energy costs" when the relevant document discusses "HVAC retrofit" and "LED conversion" without ever mentioning "energy" or "cost."
Why the gap is narrow in practice:
getDocumentDetails lets the AI explore individual documents for terminologyRevisit conditions: Consider if (a) the corpus grows beyond ~10,000 documents, (b) users report frequent "can't find it" scenarios despite AI expansion, or (c) Lucene's KNN API matures to the point where adding vectors is trivially simple.
Decision: Rejected (single-field stemming only — multi-field approach is now a candidate, see Tier 2 item E)
Reasoning: Stemming the content field directly was rejected because it causes highlighting offset issues (partial-word highlights), requires asymmetric index/search analyzers, and conflates exact and stemmed matches in scoring. The AI's wildcard + OR approach partially compensates but misses many German morphological variants (e.g., vertrag* does not match "Verträge" → "vertrage" after ICU folding).
Revised approach: Per-language stemmed shadow fields (content_stemmed_de, content_stemmed_en) — NOT stored, no term vectors, search only — combined with weighted OR queries at query time. Highlighting stays on the unstemmed content field. This follows the existing content_reversed pattern and avoids all original objections. See improvement candidate E for full details.
Decision: Deferred Reasoning: STDIO is required for Claude Desktop. Adding SSE would broaden compatibility but adds security considerations (authentication, CORS) for no immediate benefit. Revisit if browser-based MCP clients become mainstream.
Decision: Rejected Reasoning: Out of scope for a document search server. Lucene supports geo queries, but document collections rarely have meaningful geospatial data. Would add complexity for minimal benefit.
If needed: Use filters on location metadata fields instead.
Decision: Rejected (handled by MCP client) Reasoning: MCP server runs locally via STDIO - security is handled by:
Adding server-side auth would be redundant and complex for local STDIO use case.
Decision: Rejected Reasoning:
Alternative: AI-driven query refinement based on conversation context.
Decision: Deferred Reasoning:
pauseCrawler, resumeCrawler, getCrawlerStatus into single manageCrawler toolRevisit if: MCP protocol adds native tool categorization/namespacing.
When implementing any improvement from this roadmap:
SCHEMA_VERSION if any indexed field changes (see DocumentIndexer.java)fromMap() factories, success()/error() static methodsWhen implementing debugging or profiling tools (like profileQuery):
Make expensive operations opt-in — Default behavior should be fast (<10ms); deep analysis should require explicit parameter (e.g., analyzeFilterImpact=true)
Avoid exposing Lucene internals — Parse internal structures (like Explanation.toString()) into semantic, version-independent DTOs. Don't expose raw Lucene API objects.
Focus on LLM/human readability:
Document performance costs — Be explicit about execution time and number of queries executed for each analysis level
Structure for composability — Profiling data should be usable by LLMs to generate insights, not just displayed to users
Accept Lucene limitations gracefully — Document what cannot be explained (automaton internals, non-matches, passage scoring) rather than trying to work around fundamental limitations
Consider versioning — If depending on Lucene-version-specific behavior, isolate it so future upgrades are easier
When adding detailed documentation to avoid context pollution:
Keep tool descriptions concise — 1-3 sentences max; focus on what, when, and critical warnings
Parameter descriptions minimal — Type and purpose only, no examples or edge cases
Create MCP Resources for details — Comprehensive guides with examples, syntax tables, troubleshooting
Use consistent URIs — lucene://docs/{topic} for documentation, lucene://admin/{feature} for admin UIs
Structure resources as markdown — Easy to read, LLM-friendly, supports tables and code blocks
Include navigation — Table of contents, clear sections, cross-references to related resources
Test discoverability — Ensure LLM can find resources via tool descriptions referencing them
| Item | Impact | Notes |
|---|---|---|
| Structured Multi-Filter Support | High | Tier 1 #1 |
| Sort Options | Medium-High | Tier 1 #4 |
| Multi-Language Snowball Stemming | High | Tier 2 #E — SCHEMA_VERSION 3, DE+EN shadow fields |
| Date-Friendly Query Parameters | Medium | Tier 2 #6 (via filters) |
| Expanded File Format Support | Low-Medium | Tier 2 #8 |
| Markdown Bold Highlighting | Medium | New #8.5 — Changed from <em> to ** for Claude Desktop rendering |
| Query Profiling & Debugging | High | Tier 2 #9 |
| Context Pollution Reduction | High | New #10 |
| Automatic Phrase Proximity Expansion | Medium-High | New #11 — Auto-expands "Domain Design" to include variations |
| Item | Effort | Impact | Justification |
|---|---|---|---|
Tier 1 #2 - Done: suggestTerms + getTopTerms | |||
| Index Backup & Restore | Medium | Critical | Missing Gap A - Production necessity |
| Duplicate Detection | Medium | High | Missing Gap B - Common user need |
| Item | Effort | Impact | Justification |
|---|---|---|---|
| Irregular Verb Stemming | E2 - Superseded by OpenNLP lemmatization | ||
| Document Chunking | High | High | Tier 1 #3 - Solves long doc problem |
| "More Like This" | Medium | Medium | Tier 2 #5 - AI-friendly feature |
| OCR Support | Medium-High | Medium | Tier 2 #7 - Depends on user docs |
| Batch Operations | Medium | Medium | Missing Gap C - Efficiency |
| Export/Reports | Medium | Medium | Missing Gap D - Common request |
| Query Autocomplete | Medium | Medium | Missing Gap F - UX enhancement |
| Item | Priority | Notes |
|---|---|---|
| "Why Didn't This Match?" | Low-Medium | Tier 3 #11 - Complements profileQuery |
| Query Comparison Tool | Low-Medium | Tier 3 #12 - A/B testing |
| Passage-Level Scoring | Low-Medium | Tier 3 #13 - Deep debugging |
| Enhanced Optimization | Low | Tier 3 #14 - Incremental improvement |
| Query Builder | Low | Tier 3 #15 - Syntax assistance |
| Saved Searches | Low | Tier 3 #16 - Requires state |
| Search History | Low | Tier 3 #17 - Requires state |
| Document Versioning | Low | Missing Gap I - Complex |
| Streaming Results | Low | Missing Gap J - MCP limitation |
| Regex Search | Low | Missing Gap K - Documentation gap |
Based on impact, effort, and dependencies:
profileQueryThe MCP Lucene Server has matured from a basic search tool to a comprehensive, production-ready search platform with:
Key architectural strengths to preserve:
profileQuery)Desktop-first architecture decisions:
Next recommended focus: Index backup/restore (critical) + Index observability tools (high ROI) + Duplicate detection (common user need)