Search Quality Evaluation

Evaluating Search Quality

An information-retrieval (IR) system takes a query and returns a ranked list. Evaluating it means measuring whether it returns the right items in the right order for real queries — document search, product search, ticket or code search, and retrieval for RAG.

"Correct" here is ranking quality, not a single value, and live systems shift under you. So you score against a fixed, labeled test set offline, then confirm the wins hold in production. This skill covers the metrics, the labeling workflow, and how to read the scores.

The two evaluation modes

• Offline — score a retrieval pipeline against a fixed, labeled test set, before you ship. Repeatable and cheap, so it is where you iterate. It misleads only when the test set drifts from production (stale labels, different documents, different chunking or filters).
• Online — measure production behavior: reformulations, clicks, dwell, task completion. It catches the gap between "relevant" and "useful."

They often disagree. Treat offline as the gate for correctness and online as the gate for product impact. (Diagnosing a disagreement is a signal, not noise — see the last section.)

Offline ranking metrics

Build a relevance-judgment set (qrels): for each query, a human label on each candidate item. Labels are binary (relevant / not) or graded (e.g. exact / substitute / irrelevant). Then run the system and compute:

Notes that change how you read them:

• Precision and recall trade off. More candidates raises recall and lowers precision; tighter filtering does the reverse. Pick which error hurts your product more.
• MRR rewards finding one right thing fast (navigational intent), but hides missing supporting context — a system can score well on MRR while burying everything after the first hit.
• MAP rewards "relevant early" across the whole list (informational intent), but turns brittle when judgments are incomplete: unlabeled-but-retrieved items are counted as misses.
• NDCG discounts lower positions (top ranks matter more) and normalizes against the ideal ordering, so scores compare across queries with different numbers of relevant items. It is the default whenever relevance is not binary.

Choose the metric that matches the job

The metric encodes what "failure" means, so choose it from user intent, not habit.

Report a small set, name the one you optimized for, and defend a change against that number.

The evaluation workflow

1. Fix the retrieval unit. Decide whether you retrieve documents, sections, or chunks — labels must be at the same unit.
2. Build a query set. Curate common queries, hard queries, and queries that stand in for important business workflows. Derive them from real logs where you can.
3. Build a candidate set by pooling. For each query, union the top results from several systems or variants, then label that pool — so labeling doesn't bias toward one approach and unlabeled good results don't silently become "misses."
4. Write the relevance rules first, then label. Define what "relevant" means up front so different raters agree. Store the query–item labels as qrels.
5. Run and score. Run each variant through one harness and compute the metrics. Now you can compare embedding models, rerankers, filters, and chunking under identical conditions.
6. Confirm online. Ship the offline winner behind a measurement and watch the behavior signals.

For RAG, retrieval quality caps generation quality, so evaluate retrieval first, then whether the generator grounded itself in the retrieved context. Label more strictly: "supports the answer," not "related to the topic" — weak-but-topical context still raises hallucination risk.

Preprocessing is part of retrieval

What you can retrieve is only what made it into the index. Preprocessing failures show up as metric regressions even when the retriever is sound — so re-run the same harness whenever you change extraction, structure handling, chunking, or metadata.

From scores to decisions

A metric is useful only when it points to a next action:

• Inspect per-query, not just the average. Find which query classes improved, regressed, or stayed brittle. Production readiness is stability across classes, not one aggregate number.
• Separate retrieval gaps from ranking gaps. Were relevant items missing (retrieval gap) or present but buried (ranking gap)? The fixes differ.
• Make trade-offs explicit. State when higher recall is worth lower precision, or a better first result is worth worse tail coverage.
• Anchor to a baseline the team agrees is "known behavior" — a keyword system, the current production version, or a frozen config — so improvements are honest.
• Read a disagreement as a diagnosis:
  • Offline up, online down → label or intent mismatch: your qrels captured topical relevance, but users want task relevance and authority.
  • Online up, offline down → incomplete judgments: you're retrieving useful items no one labeled. Fix the dataset, not the system.

No labeled data yet?

Bootstrap. Pull a query set from behavior logs, label a small pooled candidate set with consistent rules, and grow coverage as repeated query patterns appear. Binary or dense/sparse/hybrid retrieval doesn't change the method — the work is defining relevance to match how your product uses similarity.

Source

Adapted from Evaluating Search Quality in Information Retrieval Systems, Unstructured (Apr 2026): https://unstructured.io/insights/evaluating-search-quality-in-ir-systems. The original frames preprocessing around Unstructured's product; this skill keeps the vendor-neutral IR-evaluation substance.