Quoter Eval

quoter-eval

QuoteR: A Benchmark of Quote Recommendation for Writing — Qi et al. (2022) (arXiv:2202.13145, 2022)

What this evaluates

Evaluates a model's ability to recommend relevant quotes for a given writing context. It probes cross-lingual recommendation capabilities across English, Standard Chinese, and Classical Chinese by measuring how accurately and highly a model ranks the correct quote among a pool of candidates.

Datasets

• QuoteR — total ?; splits: (unstated); repo https://github.com/thunlp/QuoteR

Metrics

• MRR (primary) — range: [0, 1]
  • Mean Reciprocal Rank: the average of 1/rank(gold) across all query contexts. Higher is better.
• NDCG@5 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at K=5. Computed as sum_{i=1}^5 (2^{r(i)}-1)/log2(i+1) with normalization constant Z_5=1, where r(i)=1 if the i-th ranked quote is gold, else 0. Paper specifically reports NDCG@5.
• Recall@K — range: [0, 1]
  • Proportion of query contexts whose gold quote appears in the top K ranked candidates. Evaluated at K=1, 10, and 100.

Input / output format

Input: A query context (left context) and a candidate set of quotes.

Output: A ranked list of candidate quotes or relevance scores used to produce the ranking.

Scoring recipe

def compute_metrics(predictions, golds):
    mrr, ndcg5, recalls = 0.0, 0.0, {k: 0.0 for k in [1, 10, 100]}
    for pred_list, gold in zip(predictions, golds):
        rank = pred_list.index(gold) + 1
        mrr += 1.0 / rank
        if rank <= 5:
            ndcg5 += 1.0 / math.log2(rank + 1)
        for k in [1, 10, 100]:
            if rank <= k:
                recalls[k] += 1.0
    n = len(predictions)
    return {
        'MRR': mrr / n,
        'NDCG@5': ndcg5 / n,
        **{f'Recall@{k}': recalls[k] / n for k in [1, 10, 100]}
    }

Common pitfalls

• The paper specifically reports NDCG@5, not the more common NDCG@10 or NDCG@20.
• Rank-based metrics (Median Rank, Mean Rank, Rank Variance) are minimized, whereas MRR, NDCG, and Recall are maximized; confusing the optimization direction leads to incorrect model selection.
• Models must be evaluated separately on the English, Standard Chinese, and Classical Chinese splits, as Chinese BERT is pre-trained on standard Chinese and performs poorly on classical Chinese without adaptation.

Evidence (verbatim from paper)

Following previous work (Ahn et al., 2016; Tan et al., 2018), we use three evaluation metrics: (1) Mean reciprocal rank (MRR), the average reciprocal values of the ranks of the gold quotes; (2) Normalized discounted cumulative gain (NDCG@K) (Järvelin and Kekäläinen, 2002), a widely used measure of ranking quality and is computed by ... We report the average of NDCG@5 scores of all the evaluated query contexts. (3) Recall@K, the proportion of query contexts whose gold quotes are ranked in respective top K candidate quotes, K={1,10,100}.

Citation

@misc{qi2022quoter,
  title={QuoteR: A Benchmark of Quote Recommendation for Writing},
  author={Qi et al. (2022)},
  year={2022},
  note={arXiv:2202.13145}
}

• arXiv: 2202.13145