First Reranking Eval

first-reranking-eval

An Early FIRST Reproduction and Improvements to Single-Token Decoding for Fast Listwise Reranking — Chen et al. (2024) (arXiv:2411.05508, 2024)

What this evaluates

Evaluates the ranking effectiveness and inference latency of single-token decoding for listwise document reranking. It probes whether using only the first-token logits of alphabetical identifiers can accurately rank candidate documents compared to full sequence generation or traditional language modeling objectives.

Datasets

• TREC DL19-22 — total ?; splits: test (-1)
• BEIR — total ?; splits: test (-1)
• MS MARCO — total ?; splits: test (-1)

Metrics

• ranking effectiveness (primary) — range: [0, 1]
  • Standard listwise ranking metrics (e.g., NDCG@10, MRR) computed over the predicted document order derived from first-token logits.
• inference latency — range: ms
  • Time taken to generate first-token logits for all candidate documents per query, measured in milliseconds using NVIDIA Nsight Systems.

Input / output format

Input: Query string concatenated with a list of top-100 candidate documents, each prefixed with a unique alphabetical identifier (A, B, C, ...).

Output: First-token logits for each alphabetical identifier, used to infer the relative ranking of the candidate documents.

Scoring recipe

def score_rerank(logits, gold_labels):
    # logits: first-token logits for identifiers A, B, C...
    # gold_labels: ground-truth relevance scores for each document
    predicted_order = np.argsort(logits)[::-1]
    # Compute standard IR metrics (e.g., NDCG@10) on predicted order vs gold labels
    return compute_ndcg_at_k(gold_labels, predicted_order, k=10)

Common pitfalls

• Using numerical identifiers instead of alphabetical ones breaks the single-token decoding constraint and invalidates the FIRST objective.
• Evaluating full sequence generation instead of first-token logits inflates latency and defeats the purpose of the single-token approach.
• Mixing up MS MARCO v1 and v2 corpora when evaluating TREC DL19-22, as they have different passage counts and indexing schemes.

Evidence (verbatim from paper)

To quantify the computational efficiency gains of FIRST’s single-token approach versus full sequence generation, we measured inference latency using NVIDIA Nsight Systems across the TREC DL19–22 datasets. We evaluated on the same data used in the original FIRST study, which comprises of several subsets of BEIR (Thakur et al., [2021]) and MS MARCO (Bajaj et al., [2018]).

Citation

@misc{chen2024first,
  title={An Early FIRST Reproduction and Improvements to Single-Token Decoding for Fast Listwise Reranking},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2411.05508}
}

• arXiv: 2411.05508