dvcrn Eidolon Search
FTS5-based memory search for AI Agents. Index markdown files once, search with 90%+ token savings.
Quick Start
1. Index memory files (once)
python3 scripts/build-index.py <memory_dir> <db_path>
Example:
python3 scripts/build-index.py ./memory ./memory.db
This creates a SQLite database with FTS5 full-text index of all .md files in the directory (recursive).
2. Search
python3 scripts/search.py <query> [limit] [db_path]
Example:
python3 scripts/search.py "Physical AI roadmap" 5
python3 scripts/search.py "project timeline" 10 ./memory.db
Default limit: 10. Default db_path: ./memory.db
Output: matching snippets with file paths and relevance scores.
3. Re-index when files change
Run build-index.py again. It rebuilds the index from scratch (fast, <1 second for typical workspaces).
When to Use
- Memory search: Find specific information across many daily notes or memory files
- Token-limited contexts: When reading all files would exceed context limits
- Repeated searches: Index once, search many times
- Large workspaces: 10+ markdown files with cumulative size >50KB
When NOT to Use
- Single small file (<5KB): just read it directly
- Need semantic/meaning-based search: FTS5 is keyword-based only
- See references/QUALITY.md for known limitations
Search Tips for Agents
FTS5 is keyword-based. Improve results by:
- Use specific terms: "Jetson Orin" not "hardware plans"
- Use OR for synonyms: "car OR vehicle OR automobile"
- Use quotes for phrases:
"Physical AI" - Try multiple queries if first attempt returns nothing
- Check file paths in results to read full context when needed
Benchmarks
- Token savings: 90%+ (measured 93-98.9%)
- Speed: 15x faster (measured 10-20x)
- Details: references/PERFORMANCE.md
Run benchmarks yourself:
python3 scripts/benchmark-recall.py # Recall@5, Recall@10
python3 scripts/benchmark-cache.py # Warm vs cold cache
DB Schema
CREATE VIRTUAL TABLE memory_fts USING fts5(path, content);
Direct SQL access:
sqlite3 memory.db "SELECT path, snippet(memory_fts, 1, '<b>', '</b>', '...', 32) FROM memory_fts WHERE memory_fts MATCH 'query' ORDER BY rank LIMIT 5;"