Skill Bm25 Index Tool

When to use

• Search large document collections (knowledge bases, notes, code)
• Index files with glob patterns for instant full-text search
• Query multiple indices simultaneously with merge strategies
• Find related documents using TF-IDF similarity
• Track search history and analyze query patterns
• Process batch queries efficiently with parallel execution

bm25-index-tool Skill

Purpose

This skill provides comprehensive access to bm25-index-tool, a production-ready CLI for lightning-fast full-text search using the BM25 ranking algorithm. Designed for AI agents and developers who need powerful search with minimal setup.

Core Capabilities:

• Index Creation: Index thousands of files in seconds with glob patterns
• Search: Query single or multiple indices with BM25 ranking
• Path Filtering: Scope searches to specific directories with glob patterns
• Related Documents: Find similar content using TF-IDF
• Batch Processing: Process multiple queries with parallel execution
• Query History: Track all searches in SQLite database
• Statistics: Analyze index health, vocabulary, term frequencies
• Caching: LRU cache for repeated queries
• Merge Strategies: Combine multi-index results (RRF, union, intersection, weighted)

When to Use This Skill

Use this skill when:

• Searching knowledge bases, notes, documentation, or code
• Building search features in applications
• Analyzing document collections with BM25 ranking
• Finding related content based on similarity
• Processing batch queries for efficiency
• Working with Obsidian vaults, documentation sites, or code repositories
• Needing JSON output for downstream processing

Do NOT use this skill for:

• Tasks requiring real-time indexing (index rebuild required)
• Incremental updates (full re-index only)
• Fuzzy matching (BM25 is keyword-based)
• Vector search or semantic similarity (use embeddings instead)

CLI Tool: bm25-index-tool

The bm25-index-tool provides lightning-fast BM25 search with advanced features like multi-index queries, path filtering, related documents, and comprehensive history tracking.

Installation

# Clone and install
git clone https://github.com/dnvriend/bm25-index-tool.git
cd bm25-index-tool
uv tool install .

Prerequisites

• Python 3.14+
• uv package manager

Quick Start

# Create an index from markdown files
bm25-index-tool create vault --pattern "~/vault/**/*.md"

# Search the index
bm25-index-tool query vault "kubernetes networking"

# Search with JSON output (AI agent friendly)
bm25-index-tool query vault "docker" --format json

# Get comprehensive help with examples
bm25-index-tool query --help  # Shows 20+ examples

Progressive Disclosure

help - Show Help Information

Display help information for CLI commands.

Usage:

bm25-index-tool --help
bm25-index-tool COMMAND --help

Examples:

# General help
bm25-index-tool --help

# Command help
bm25-index-tool command --help

# Version info
bm25-index-tool --version

Multi-Level Verbosity Logging

Control logging detail with progressive verbosity levels. All logs output to stderr.

Logging Levels:

Examples:

# INFO level - see operations
bm25-index-tool command -v

# DEBUG level - see detailed info
bm25-index-tool command -vv

# TRACE level - see all internals
bm25-index-tool command -vvv

Shell Completion

Native shell completion for bash, zsh, and fish.

Installation:

# Bash (add to ~/.bashrc)
eval "$(bm25-index-tool completion bash)"

# Zsh (add to ~/.zshrc)
eval "$(bm25-index-tool completion zsh)"

# Fish (save to completions)
bm25-index-tool completion fish > ~/.config/fish/completions/bm25-index-tool.fish

Pipeline Composition

Compose commands with Unix pipes for powerful workflows.

Examples:

# Example pipeline workflows will be added when CLI commands are implemented
bm25-index-tool command --json | jq '.'

Common Issues

Issue: Command not found

# Verify installation
bm25-index-tool --version

# Reinstall if needed
cd bm25-index-tool
uv tool install . --reinstall

Issue: General errors

• Try with verbose flag: -vv to see detailed error information
• Check that all prerequisites are installed
• Ensure you're using Python 3.14+

Getting Help

# Show help
bm25-index-tool --help

# Command-specific help
bm25-index-tool COMMAND --help

Exit Codes

• 0: Success
• 1: Client error (invalid arguments, validation failed)
• 2: Server error (API error, network issue)
• 3: Network error (connection failed, timeout)

Output Formats

Default Output:

• Human-readable formatted output
• Varies by command

JSON Output (--json flag):

• Machine-readable structured data
• Perfect for pipelines and processing
• Available on commands that support structured output

Best Practices

1. Use verbosity progressively: Start with -v, increase to -vv/-vvv only if needed
2. Check help first: Use --help to understand command options
3. Leverage shell completion: Install completion for better CLI experience

Resources

• GitHub: https://github.com/dnvriend/bm25-index-tool
• Python Package Index: https://pypi.org/project/bm25-index-tool/
• Documentation: