Qrels Construction

Build and audit per-question relevance judgments for a projected benchmark - join a reviewer workbook with verified questions, judge candidate documents against hop standards, expand the pool over exact and near duplicates, and produce the distribution figures. Use when asked to build qrels, finalize relevance judgments, expand judgments over duplicates, check how redundant a judged pool is, or make qrels figures. Triggers include "qrels", "relevance judgments", "judge documents", "duplicate expansion", "near duplicate qrels", "how many relevant documents per question".

castorini Updated
Projecting QA benchmarks onto ClimbMix

File contents

castorini/cmass commit 4f25af4669

Frequently asked questions

npx get-skill add castorini/qrels-construction