# Chroma Hybrid Search > Performs high-precision hybrid retrieval (BM25 + ChromaDB vector search + BGE-Reranker semantic re-ranking) over the local hot store (knowledge-base/, experience/) and cold store (cold-notes/raw.jsonl). Use this skill when high-accuracy code or solution retrieval is needed and AI hallucination must be minimized. Running for the first time in a directory will automatically trigger initialization. Typically invoked by deep-memory's knowledge-base retrieval step when keyword matching is insufficient, but can also be called directly for ad-hoc searches. - Skill: `kevintsai1202/chroma-hybrid-search` (Agent Skill, multi-file: 10 files) - Install (CLI): `npx skillmds@latest add kevintsai1202/chroma-hybrid-search` - Raw SKILL.md: https://api.skillmd.com/api/skills/kevintsai1202/chroma-hybrid-search/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: kevintsai1202 (https://skillmd.com/u/kevintsai1202) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/kevintsai1202/chroma-hybrid-search --- # Chroma Hybrid Search Skill This skill provides local RAG retrieval by combining vector search (semantic) with BM25 keyword search (exact match), followed by Cross-Encoder re-ranking using the BGE-Reranker model. It searches both the hot store (`knowledge-base/*.md`, `experience/*.md`) and the cold store (`cold-notes/raw.jsonl`) — the document set it reads must match what `update_db.py` indexed, or vectorized cold-store entries would be dropped from results. > **Cross-platform command convention** — in every command below, `` is the virtual-env Python: > > - **Windows (PowerShell):** `& "$HOME\.deep-memory\.venv\Scripts\python"` > - **Linux / macOS:** `~/.deep-memory/.venv/bin/python` > > The venv lives at a fixed absolute path inside the global workspace (`~/.deep-memory/.venv`), next to the data — commands work from any project directory; no project-local `.venv` is assumed. On PowerShell keep the `&` call operator and `$HOME` (PowerShell does not expand `~` in arguments to native commands); in Git Bash on Windows the same venv is `~/.deep-memory/.venv/Scripts/python`. > > > All `skills/...` paths assume the skill pack lives inside your current project (project-local install). > > **Workspace Storage Path Resolution:** > By default, deep-memory uses the user's global directory `~/.deep-memory` (which resolves to `C:\Users\\.deep-memory` on Windows) to store all knowledge bases, cold notes, and database files. This unifies memories across all your project workspaces. > - If you want to use a specific directory, set the `DEEP_MEMORY_WORKSPACE` environment variable (e.g., `DEEP_MEMORY_WORKSPACE="."` or `DEEP_MEMORY_WORKSPACE="D:\my-memories"`). > - You can also pass `--workspace ` to any script to override the workspace path for that specific command. ## ⚙️ First-Time Initialization (Bootstrap) When this skill is first invoked by the Agent or user, confirm that the `.venv` virtual environment and the `chroma_hybrid_db` vector database have been created and all required packages are installed. If not, execute the following steps: ```bash # 1. Create virtual environment (if it doesn't exist) — fixed location in the global workspace, shared by all projects # Windows (PowerShell): python -m venv "$HOME\.deep-memory\.venv" # Linux / macOS: python3 -m venv ~/.deep-memory/.venv # 2. Install required packages (no activation needed — call the venv Python directly) -m pip install -r skills/chroma-hybrid-search/requirements.txt # 3. Initialize and build the local vector index database skills/chroma-hybrid-search/scripts/update_db.py ``` > **Notes:** > 1. Do NOT commit `.venv` or `chroma_hybrid_db/` to GitHub — both live under `~/.deep-memory` (outside any repo) and must be generated locally using the steps above. > 2. Step 3 is only needed once, to create the index. After that `search.py` tops it up automatically before every query — see Index Maintenance below. --- ## 🔄 Index Maintenance `search.py` tops up the index before every query, so **`update_db.py` is no longer part of the daily loop**. An indexing failure prints a warning and never blocks the search; `--no-auto-index` disables the behaviour. Run `update_db.py` explicitly only to index new entries immediately rather than at the next search — e.g. right after a refinement pass. `update_db.py --rebuild` forces every entry to be re-embedded (~6 minutes for a 1,265-entry store); it is only for making vectors correspond exactly to the latest text format, never required for normal operation. ### Content-addressed IDs Cold-store entries are keyed by a semantic hash — `cold-notes#` — not by line number. The hash covers `topic` / `content` / `tags` / `skill` / `memory_type` and deliberately excludes `date` / `time` / `project` / `quality` and every template string. So: - Deleting or inserting a line anywhere in `raw.jsonl` does **not** re-embed the entries after it. Measured on a 928-line store: removing the middle line re-embedded **0** entries — the 463 entries after it only had their `line` metadata refreshed, at zero embedding cost. - Marking an entry `quality: reviewed` during refinement does **not** trigger a re-embed. - Changing how the indexed text is assembled does **not** invalidate the store. The line number survives in metadata as `line` and is returned with every cold-store hit, so results stay traceable to their source line. ### Run log Each indexing run appends one JSON line to `~/.deep-memory/logs/index-runs.jsonl` with per-stage timings and why entries were re-embedded: | Reason | Meaning | |---|---| | `new` | Genuinely a new entry — normal | | `text-changed` | The entry's content actually changed | | `id-missing` | An old ID vanished as a new one appeared — the signature of an ID shift or a version mismatch | A run that re-embeds more than 20% of the store prints a warning with the reason breakdown. When the reason is `id-missing`, the usual cause is that the repo copy and the `~/.claude/skills` installed copy of `kb_reader.py` are on different versions — they assemble the indexed text differently, so each sees the other's entries as changed. --- ## 🔍 Usage The Agent can execute `scripts/search.py` directly from the terminal to perform precise retrieval over the knowledge base and experience store. ### 1. Hybrid Search + Semantic Re-ranking (Default — Recommended) Best for complex questions requiring deep semantic understanding: ```bash skills/chroma-hybrid-search/scripts/search.py --query "spring animation tuning guidelines" --min-score 0.35 ``` Always pass `--min-score 0.35` in this mode — besides filtering noise, it lets the project-first pass fall back to the full cross-project store when the current project only has low-relevance hits (without it, a near-zero-score project hit still counts as "found something" and blocks the fallback). ### 2. Vector Similarity Search Only Best for cross-language or fuzzy semantic search: ```bash skills/chroma-hybrid-search/scripts/search.py --query "Spring Boot 404" --mode vector ``` ### 3. BM25 Keyword Search Only Best for finding exact proper nouns, variable names, or code snippets: ```bash skills/chroma-hybrid-search/scripts/search.py --query "pxmin pxmax" --mode bm25 ``` ### 4. Scoped to One Skill, Tag, or Memory Type Cold-store entries carry `skill`, `tags`, and `memory_type` metadata; `experience/skill-[skill-id].md` files carry `skill` derived from their filename, and hot-store entries are tagged as `knowledge` or `experience` by source folder. Use `--skill` for an exact skill-id match, `--tag` to check the tags array (Chroma's native `$contains`, requires chromadb ≥1.5.0 — already pinned in requirements.txt), and `--memory-type` to separate general knowledge from skill/tool experience. These filters narrow the BM25 corpus and the vector `where` clause identically, so every retrieval method sees the same filtered candidate set: ```bash skills/chroma-hybrid-search/scripts/search.py --query "session timeout" --skill backend-dev skills/chroma-hybrid-search/scripts/search.py --query "config drift" --tag redis skills/chroma-hybrid-search/scripts/search.py --query "agent browser fill fallback" --memory-type experience ``` Plain `knowledge-base/*.md` category files have no single skill/tag (they mix many topics), so they're never included by `--skill` or `--tag` filters unless the indexed entry explicitly has that metadata; they do participate in `--memory-type knowledge`. --- ## 📋 Output Format The script outputs a standard JSON array. `path` for hot-store hits is `file.md#entry-slug` — one `## 🔧` entry, not the whole file (see "Entry-Level Chunking" below). **Use the `text` field directly as context — do not separately open the file at `path` with the Read tool.** Stripping the `#entry-slug` and reading the whole source file re-introduces exactly the whole-file dilution chunking exists to avoid; `path` is for citation (telling the user where a fact came from), not an instruction to read further. ```json [ { "path": "experience/skill-remotion-best-practices.md#fps-30-causes-av-desync", "rerank_score": 0.9402, "text": "## 🔧 FPS 30 causes A/V desync\n**Date:** ...\n(just this one entry, not the rest of the file)" } ] ``` ## ✂️ Entry-Level Chunking `knowledge-base/*.md` and `experience/*.md` are indexed per `## 🔧` entry, not per file — `kb_reader.py`'s `split_entries()` splits on that heading and gives each entry a stable slug (from its title, de-duplicated within the file) so results point to the specific matching entry. A file with no `## 🔧` headings at all falls back to whole-file indexing for backward compatibility. `update_db.py` and `search.py` both import this splitting logic from the same `kb_reader.py` — if you ever need to change how entries are parsed, change it there once, not in both scripts. ## 🛡️ Anti-Hallucination RAG Routing Guide - **Filename-first matching**: If the user's question explicitly mentions a filename, read that file's full content directly via Python `glob` or fast text search first. - **Tiered RAG**: Prioritize filename matching; fall back to `search.py` hybrid search + rerank only if no match is found. - **Score threshold filtering**: Do not pass any chunks or files with a Rerank Score below `0.35` to the model — this prevents noise from degrading response quality. Apply it search-side with `--min-score 0.35` rather than filtering the output afterwards: the search-side filter is also what allows the project-first pass to fall back to the cross-project store when project hits are all low-relevance.