Embeddings Search

embeddings-search — make and judge the vectors

You own the embedding technique layer: turn a corpus into searchable vectors, turn a question into a good retrieval, and measure whether that retrieval is any good. You stop the moment the right chunks come back, measured by a number. You do not assemble a prompt or generate an answer.

Route the adjacent surfaces away:

• Operating the store — collection schema, HNSW/IVFFlat tuning, metadata-filter path, quantization, ef_search recall knobs → ../vector-db/SKILL.md. You decide what vectors go in and how to query; vector-db decides how the store holds and serves them.
• The full retrieve → rerank → prompt → generate → answer loop and its groundedness / faithfulness eval → ../rag/SKILL.md.
• Pulling typed fields out of documents (invoice number, date, total) → ../structured-extraction/SKILL.md.
• Writing the prompt the model reasons with → ../prompt-engineering/SKILL.md.

1. Pick the embedding model

Decide on three axes: language coverage, quality tier (read MTEB but don't worship it), and cost — where cost is set by dimensions, because dims set storage and memory.

Quality anchor (mid-2026 MTEB English retrieval): Gemini ~68.3, Cohere embed-v4 ~65.2, OpenAI 3-large ~64.6, BGE-M3 ~63.0. MTEB is the standard comparison, not a verdict on your domain — references/models.md carries the full model matrix (dims, max tokens, price, input_type convention, Matryoshka support) and how to read MTEB without over-trusting it.

Two hard rules — each is a silent failure, no error, just worse results:

• Match the distance metric to the model. A cosine-trained model indexed or queried with L2 ranks silently wrong. Cosine → <=> in pgvector / Distance.COSINE in Qdrant. The index operator itself is vector-db's job; the requirement originates from the model, so state it in your config.
• Respect query/document asymmetry. Cohere, Voyage, Gemini, e5, BGE expect a different prompt or input_type for the query vs the stored passage. Embed both sides identically and recall silently drops.

Dimensions = cost. A 1024-dim float32 vector is 4 KB; at 10M docs that is 40 GB, and doubling dims doubles storage and memory. Matryoshka-trained models (OpenAI 3-*, Cohere, Gemini) let you truncate dims for graceful degradation — never re-embed the whole corpus just to shrink vectors.

2. Chunk the corpus

Start boring. Upgrade only when a number tells you to.

# Default recipe: recursive split, TOKEN-accurate count, 10–20% overlap.
import tiktoken
from langchain_text_splitters import RecursiveCharacterTextSplitter

enc = tiktoken.get_encoding("cl100k_base")
splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    encoding_name="cl100k_base",
    chunk_size=512,      # tokens, not characters
    chunk_overlap=64,    # ~12% — keeps sentences from being cut mid-thought
)
chunks = splitter.split_text(document_text)

Counting by characters instead of tokens is the most common own-goal: 512 characters is ~100–130 tokens of English and far fewer of CJK, so "512" silently means different things per language and per model limit.

Upgrade ladder — graduate only when retrieval metrics (section 6) justify the added compute:

Embed the searchable text; store the rest as metadata. What you embed is what gets matched — don't bury the answer text under boilerplate, and don't embed raw HTML.

3. Construct the query

The query is half the retrieval. Embed it the way the model expects, then improve it only when recall data says you should.

# Asymmetric model: query and document use DIFFERENT input_type. Getting this wrong is silent.
q_vec   = embed(text=user_question, input_type="search_query")     # Cohere / Voyage
d_vec   = embed(text=passage,       input_type="search_document")
# e5 / BGE convention is a textual prefix instead:
#   query    -> "query: how do refunds work"
#   passage  -> "passage: Refunds are processed within 14 days…"

Query-side techniques, when each pays:

• Query rewriting — when user queries are terse or full of pronouns; normalise before embedding.
• HyDE (embed a hypothetical answer, not the question) — when questions are short and answers are long/technical, so the answer-shaped vector lands nearer the passage.
• Multi-query (fan out 3–4 paraphrases, union the hits) — when one phrasing under-recalls; costs N embeds and a dedup.

4. Hybrid + rerank

Dense and sparse fail in complementary ways: BM25 nails exact terms, IDs, SKUs, rare tokens; dense nails paraphrase. That is why exact-match queries return nothing while paraphrases work — the fix is adding sparse, not a bigger embedding model.

Fuse by rank, not score, with Reciprocal Rank Fusion so you never have to calibrate BM25 tf-idf magnitudes against cosine magnitudes per corpus:

# RRF: each doc scores 1/(k + rank) summed across the dense and sparse lists. k≈60.
def rrf(*ranked_lists, k=60):
    scores = {}
    for lst in ranked_lists:                 # fan-in 20–100 per list
        for rank, doc_id in enumerate(lst):  # rank is 0-based
            scores[doc_id] = scores.get(doc_id, 0) + 1.0 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

Then a cross-encoder reranker sits AFTER fusion: take top-50, score each against the original query, keep top-5 for downstream use.

• Current models: Cohere rerank-v4.0-pro / rerank-v4.0-fast (note rerank-3.5 is deprecated). Voyage rerank-2.5 (2025-08-11) is the first widely available instruction-following reranker — 32K-token context (8× Cohere v3.5), reports +7.94% accuracy vs Cohere v3.5 on a 93-dataset suite.
• A reranker raises precision but cannot recover a doc the retriever never returned. Recall is upstream. If the right chunk isn't in the top-50, no reranker saves you — fix recall first.

The fusion and index mechanics per engine (how Qdrant/Weaviate/pgvector run hybrid) are vector-db's: ../vector-db/SKILL.md.

5. Measure retrieval quality

This is the rigor of the skill. "Search is bad" is not actionable; "recall@10 is 0.62" is.

1. Build a golden query set — 30–50+ labeled query → relevant doc ids pairs drawn from real questions. This is the asset; everything else is reproducible from it.
2. Pick metrics — definitions, a runnable Python eval skeleton, golden-set construction and the A/B-one-change methodology are in references/evaluation.md:
   • recall@k — of the truly relevant docs, how many landed in the top-k. Catches "the right chunk never came back."
   • nDCG@k — rewards relevant docs ranked higher. Catches "right docs, wrong order."
   • MRR — how high the first relevant doc sits. Catches "the one answer is buried."
3. Move-the-number loop. Establish a baseline, change exactly one thing (model OR chunk size OR fusion OR reranker), re-measure on the same query set. Two changes at once and you learn nothing.

A retrieval change shipped without a before/after number on a golden set is a guess. verify.sh flags hybrid/rerank artifacts that mention no recall/nDCG/MRR for exactly this reason.

Anti-patterns