Lexical Kb

lexical-kb — query an embedding-free knowledgebase

This KB has no semantic search and no embedding model. Retrieval is pure lexical BM25 over a precomputed inverted index. That design moves one job onto you: bridging the gap between how the user phrases a question and how the corpus phrases the answer. An embedding model would do this with a vector; here you are the semantic layer — you expand the query into terms before searching.

Corpus: {{SOURCE}} ({{CHUNK_COUNT}} chunks).

The retrieval protocol — follow every step

A raw user question fed straight to BM25 underperforms: it matches only the exact words the user happened to use. The expansion step is what makes lexical retrieval competitive with embeddings. Do not skip it.

1. Read the question. Extract core terms — the essential nouns, proper nouns, and identifiers the answer MUST contain. These carry full weight.

2. Generate expand terms — synonyms, morphological variants (plural/verb forms), acronym expansions and contractions, and adjacent concepts. These carry lower weight. This is the work the missing embedding model would have done. Be generous: 5–15 expansion terms is normal.

3. Run the searcher. It ships in this bundle. Pass the user's original question via --query AND your term groups — expansion is additive, it never replaces the user's words:

   python3 search.py \
     --query "how does centered simhash differ from random projection?" \
     --core "simhash" --core "centered" \
     --expand "random projection" --expand "hyperplane" --expand "LSH" \
     --expand "binary quantization" --expand "hamming distance" \
     --k 5
   

   --core/--expand are repeatable; pass phrases, the searcher tokenizes them. The --query terms contribute at a low floor weight so a curated synonym can lift a result but can never drop a doc the literal question would have matched. Defaults: core 1.0, expand 0.4, query-floor 0.25, top-k 5. Keep expansion targeted — terms too generic ("system", "process") leak into unrelated chunks and blur the ranking.

4. Read the returned chunks. Answer from them, and cite chunk ids inline. The chunks are the source of truth the user installed. When a chunk contradicts your prior knowledge, the chunk wins — say so. When the chunks do not contain the answer, say that plainly rather than filling the gap from memory.

When you cannot expand — RM3 fallback

If the query is outside any domain you can expand confidently, pass it raw with pseudo-relevance feedback. The searcher harvests expansion terms from the corpus's own top hits — model-free, weaker than your expansion, and prone to drift when the first pass is off-topic, so prefer real expansion when you can:

python3 search.py --query "the user's raw question" --rm3 --k 5

Metadata filtering

Each chunk carries structured meta (e.g. title, source_path, section). Filter on it with --filter (repeatable). Filtering narrows by attribute; it does not rank — combine it with term search.

python3 search.py --core "factions" --filter "section=blog" --filter "date>=2025" --k 5

Operators: =, !=, ~ (substring), >, >=, <, <= (numeric when both sides parse, else lexicographic — ISO dates sort correctly).

Output

search.py prints JSON: {"hits": [{id, score, text, meta}, ...]}, sorted by descending BM25 score. Surface the top hits to the user with their ids, then answer using them as authoritative context.

Mechanics

• Pure Python stdlib. No pip install, no model download, no network.
• The bundle is self-contained: search.py, index.json, chunks.jsonl. Run search.py from inside the bundle directory (it defaults --index to its own location) or pass --index /path/to/bundle.