# Debugging Playbook > Playbook for diagnosing Searchlite issues across CLI/HTTP, storage, and indexing; use when triaging failures or odd search results. Use when this capability is needed. - Skill: `tomevault-io/debugging-playbook` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add tomevault-io/debugging-playbook` - Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/debugging-playbook/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: tomevault-io (https://skillmd.com/u/tomevault-io) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/tomevault-io/debugging-playbook --- # Debugging Playbook ## High-level mental model Searchlite is a single-index, single-writer system with: - buffered writes (CLI or HTTP) - durability via WAL + atomic manifest updates - immutable segments + periodic compaction - “refresh” semantics for making new commits visible to readers (especially in HTTP mode) Most “weirdness” is one of: 1. **you didn’t commit** (writes are buffered) 2. **you didn’t refresh** (reader is stale) 3. **your schema isn’t configured for what you’re asking** (e.g., missing `fast` fields for filters/aggs) 4. **you’re seeing crash-window replay artifacts** (extra segments until compaction) 5. **you’re hitting HTTP limits** (body size, concurrency, timeouts) --- ## 10-minute triage checklist 1. **Identify surface area** - CLI? embedded Rust API? HTTP server? 2. **Capture exact versions + features** - git SHA/tag (if known) - build flags / enabled features (`vectors` changes behavior materially) 3. **Get a minimal reproduction** - schema JSON - the smallest NDJSON/JSON doc set that reproduces - the exact search request JSON or CLI flags 4. **Check index state** - CLI: `searchlite inspect ` and `searchlite compact ` - HTTP: `GET /inspect`, `GET /stats`, optionally `POST /compact` 5. **Confirm lifecycle steps** - Writes are queued → must call `commit` - Depending on config, may need `refresh` for immediate visibility --- ## CLI debugging moves ### A) “I added docs but search returns nothing” Common causes: - Documents missing the required ID field (`doc_id_field`, default `_id`) - You didn’t run `commit` - Query is hitting an analyzer mismatch or field mismatch Checklist: - Verify you ran: - `searchlite add …` - `searchlite commit …` - Run: - `searchlite inspect ` - `searchlite search --q "..." --return-stored` (or request JSON with `return_stored: true`) - If segments ballooned after a crash, run: - `searchlite compact ` ### B) “Filters/aggs don’t work / always empty” Filters and aggregations depend on `fast` fields. - Ensure the target keyword/numeric fields are `"fast": true` in schema. - For nested filters, ensure nested subfields are `fast` too and use `Nested` filters properly. --- ## HTTP debugging moves ### A) Safety + environment sanity HTTP server has **no auth / no rate limiting**; don’t expose it to untrusted networks. When debugging production-ish behavior, always capture: - bind addr (`--bind` / `SEARCHLITE_BIND_ADDR`) - index path (`--index` / `SEARCHLITE_INDEX_PATH`) - resource limits: - `--max-body-bytes` - `--max-concurrency` - `--request-timeout-secs` - shutdown grace - refresh behavior: - `--refresh-on-commit` ### B) Confirm lifecycle and visibility Writes issued via: - `POST /add` (NDJSON streaming) - `POST /bulk` (JSON bulk) - `POST /delete` …are queued and only become durable/searchable after: - `POST /commit` Then visibility depends on staleness needs: - if `--refresh-on-commit` is enabled, new data becomes visible immediately - otherwise call: - `POST /refresh` ### C) Use maintenance endpoints for ground truth - `POST /refresh` — reload readers - `POST /compact` — merge segments + clean up fragmentation - `GET /inspect` — manifest + segments info - `GET /stats` — doc/segment counts ### D) Interpret HTTP errors All errors return: - `{"error":{"type":"...","reason":"..."}}` If you see a 413/timeout-ish behavior: - check `--max-body-bytes` and `--request-timeout-secs` - for NDJSON streaming, reduce batch sizes / request size --- ## Ranking correctness + “why did this score happen?” Search supports explicit debugging aids: ### `explain: true` - Returns per-hit score breakdowns. - Use this for correctness (“why did doc X win?”), not for performance runs. ### `profile: true` - Adds execution stats like: - candidates examined - scored docs - postings advances - timing buckets (search vs rescore) - Use this for performance triage (“why is this query slow?”). Both are off by default for overhead reasons. --- ## “This query is slow” playbook 1. Enable `profile: true` on the request and capture the profile blob. 2. Compare execution modes: - `execution=bm25` (baseline full scoring) - `execution=wand` (default exact pruning) - `execution=bmw` (block-max pruning) 3. Confirm you’re not forcing expensive features: - rescore windows - heavy highlighting - large aggregations on high-cardinality fields 4. If using aggregations: - confirm required fields are `fast` - consider sampling knobs where supported 5. If using vector search (feature `vectors`): - remember ANN is approximate; raising `candidate_size`/`ef_search` increases recall but costs CPU --- ## Vector search “it errors / results look wrong” Common causes: - wrong vector dimension (rejected) - using cosine without normalization assumptions (Searchlite normalizes cosine vectors automatically) - expecting exact results from ANN (HNSW is approximate) Knobs to tune: - `candidate_size` (oversampling) - `ef_search` (beam width) - `vector_filter` (reduce candidate set) --- ## Nested filter “it matches too much / too little” Key rule: nested filters are evaluated per object, not across the entire document. Correct pattern: - wrap clauses under `Nested` with `path` - for deeper nesting, use a nested `Nested` inside the parent nested filter Common gotcha: - forgetting to mark nested filter fields as `fast: true` --- ## Storage/durability anomalies Symptoms: - “extra segments” after crash - “duplicate generation” feeling after restart Remember the documented crash window: - if the process dies after manifest persisted but before WAL truncation, WAL replay reapplies last batch - data isn’t lost, but you may see extra segment generations until compaction Fix: - run `compact` (CLI or HTTP) to merge and clean up --- ## What to attach to a bug report - schema JSON - small docs ndjson/json - exact request JSON (include `execution`, `profile`, `explain` if relevant) - output of `inspect` + `stats` - server flags/env (HTTP) - if durability related: note whether crash/restart occurred --- > Converted and distributed by [TomeVault](https://tomevault.io/claim/davidkelley) — claim your Tome and manage your conversions.