castorini Qrels construction
Runs over pipelines/bcp_climbmix/qrels/ in this repository. Every step is
CLI-driven, takes --work-dir, and is resumable per question.
Pick the entry point by what is being asked
| ask | run |
|---|---|
| "build the qrels" | step0 → step1 → judging loop → step4 |
| "what still needs judging for question X" | step2_render_worksheet.py --qid X |
| "expand over duplicates" | step5_expand_duplicates.py scan then the read/apply loop |
| "how far along is the expansion" | step5_expand_duplicates.py status |
| "how many relevant docs per question / per hop" | analysis/qrels_stats.py |
| "how redundant is the pool" | analysis/qrels_near_dup.py |
Inputs are environment variables (QRELS_WORKBOOK, QRELS_QUESTIONS,
QRELS_DOC_CMD, QRELS_NEAR_DUP_DIR, QRELS_EXACT_DUPS) or the matching
flags. Nothing is hardcoded; a missing input fails with the flag and variable
to set, not a stack trace.
What is released and what is not
The release is per-question relevance judgments. The hop decomposition and
the per-hop assignments are internal: they drive judging and auditing, and they
stay out of the release. Do not commit work/, out/, or any JSONL the
pipeline produces — .gitignore covers them, so check git status before
adding files rather than adding a directory wholesale.
Judging discipline
This is the part that decides quality, and it is not automatable.
- Write the hop standard before reading. What must a document state, and what near miss must it reject? A bar set while looking at a candidate drifts toward whatever the candidate happens to say.
- Read every document in full. Not the workbook excerpt, not a truncated render. The deciding sentence is frequently past the first screen, and two copies of the same page often differ exactly there.
- Record only rejections, each with the sentence that decides it and a contrast case — a document that does pass, and why. A rejection with no contrast is the one most likely to be wrong.
- Attribution is the common failure. On a page covering several entities, the sentence that would support the hop often belongs to a different subject. Check whose sentence it is before accepting.
- Precision over recall, and never relax a standard to make a hop non-empty. A kept hop with no support is reported by step 4 for review; that is the correct outcome, not a reason to lower the bar.
Two views of duplication — do not conflate them
- Affected: documents that have a near-duplicate partner in the same query's qrels. The right number for review scope: if a judged document has a twin, both need looking at.
- Removable: copies beyond one representative per cluster. The right number for characterising redundancy.
analysis/qrels_near_dup.py reports both (n_near_dup and n_redundant).
Reporting "affected" as a redundancy figure overstates it.
Things that will bite
Two document-id spellings. Corpus artifacts zero-pad the sequence
(shard_00045_06622), the retrieval API and qrels do not (shard_00045_6622).
An unnormalised join silently returns fewer duplicates than exist — no error,
just a quietly incomplete answer. Normalise on every join.
to_pylist() on a duplicate column materialises one list-of-struct row per
corpus document. Use Arrow .take() to filter first.
Inclusive integer buckets drop continuous values. 0–9 then 10–19 loses
everything between 9 and 10. Use half-open [lo, hi) and assert the buckets
account for every input.
Figures are authored at final size. They are saved on a fixed canvas, not a tight bounding box, so LaTeX does not rescale them and the point sizes hold. A tight bbox grows the canvas to fit the labels, which silently changes the scale factor and shrinks the text on the page.