Score Retrieval Quality
"Search seems to work" is a feeling, not a measurement. Retrieval is good when the right notes come back and the wrong ones do not — both measurable on a handful of labeled queries.
Procedure
- Build a labeled set of 10-20 queries you have actually needed, in
notes/retrieval-eval.jsonl. - For each query, list the note ids that should return (the gold set), by hand.
- Run each query against your retriever and capture the returned ids.
- Compute recall = |returned AND gold| / |gold| and precision = |returned AND gold| / |returned|.
- Report the mean of each; a healthy store clears recall 0.8 and precision 0.6 on this set.
- Look at the misses: a query with recall 0 usually means the note was never written, not that search failed.
- Look at the false hits: precision 0.2 usually means the query terms are too common — add a filter.
- Fix the cheapest side first: a tagging gap (see
tag-notes-for-precision-recall) is cheaper than re-embedding. - Re-run the same labeled set after any change; the set is the regression test.
- Keep the eval file in the repo so the score is reproducible, not remembered.
Pitfalls
Judging retrieval by one lucky query and declaring it good.
Counting a note as a hit because its title matched while its content was irrelevant.
Changing the retriever and re-scoring on a new query set, so nothing is comparable.
Treating recall and precision as one number, then optimizing whichever the current query favours.
Blaming the retriever when the gold note does not exist in the store at all.
Scoring on queries invented for the test rather than ones you actually ran.
Reporting a single aggregate and hiding the queries that returned nothing.
Freezing the eval and never adding new queries, so the score stops reflecting real use.
Verification
jq -s '{recall: (map(.hit)|add/length), precision: (map(.prec)|add/length)}' notes/retrieval-eval-results.json
# passes when recall >= 0.8 and precision >= 0.6 on the frozen query set
Report to the user: the query count, mean recall, mean precision, and the two worst queries with the reason.