# Quoter Eval

> Evaluates a model's ability to recommend relevant quotes for a given writing context. It probes cross-lingual recommendation capabilities across English, Standard Chinese, and Classical Chinese by measuring how accurately and highly a model ranks the correct quote among a pool of candidates. Use when the user wants to benchmark on QuoteR, or asks about evaluating this task. Reports MRR.

- Skill: `qhjqhj00/quoter-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/quoter-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/quoter-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/quoter-eval

---


# quoter-eval

> QuoteR: A Benchmark of Quote Recommendation for Writing — Qi et al. (2022) (arXiv:2202.13145, 2022)

## What this evaluates

Evaluates a model's ability to recommend relevant quotes for a given writing context. It probes cross-lingual recommendation capabilities across English, Standard Chinese, and Classical Chinese by measuring how accurately and highly a model ranks the correct quote among a pool of candidates.

## Datasets

- **QuoteR** — total ?; splits: (unstated); repo https://github.com/thunlp/QuoteR

## Metrics

- `MRR` **(primary)** — range: [0, 1]
  - Mean Reciprocal Rank: the average of 1/rank(gold) across all query contexts. Higher is better.
- `NDCG@5` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at K=5. Computed as sum_{i=1}^5 (2^{r(i)}-1)/log2(i+1) with normalization constant Z_5=1, where r(i)=1 if the i-th ranked quote is gold, else 0. Paper specifically reports NDCG@5.
- `Recall@K` — range: [0, 1]
  - Proportion of query contexts whose gold quote appears in the top K ranked candidates. Evaluated at K=1, 10, and 100.

## Input / output format

**Input**: A query context (left context) and a candidate set of quotes.

**Output**: A ranked list of candidate quotes or relevance scores used to produce the ranking.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    mrr, ndcg5, recalls = 0.0, 0.0, {k: 0.0 for k in [1, 10, 100]}
    for pred_list, gold in zip(predictions, golds):
        rank = pred_list.index(gold) + 1
        mrr += 1.0 / rank
        if rank <= 5:
            ndcg5 += 1.0 / math.log2(rank + 1)
        for k in [1, 10, 100]:
            if rank <= k:
                recalls[k] += 1.0
    n = len(predictions)
    return {
        'MRR': mrr / n,
        'NDCG@5': ndcg5 / n,
        **{f'Recall@{k}': recalls[k] / n for k in [1, 10, 100]}
    }
```

## Common pitfalls

- The paper specifically reports NDCG@5, not the more common NDCG@10 or NDCG@20.
- Rank-based metrics (Median Rank, Mean Rank, Rank Variance) are minimized, whereas MRR, NDCG, and Recall are maximized; confusing the optimization direction leads to incorrect model selection.
- Models must be evaluated separately on the English, Standard Chinese, and Classical Chinese splits, as Chinese BERT is pre-trained on standard Chinese and performs poorly on classical Chinese without adaptation.

## Evidence (verbatim from paper)

> Following previous work (Ahn et al., 2016; Tan et al., 2018), we use three evaluation metrics: (1) Mean reciprocal rank (MRR), the average reciprocal values of the ranks of the gold quotes; (2) Normalized discounted cumulative gain (NDCG@K) (Järvelin and Kekäläinen, 2002), a widely used measure of ranking quality and is computed by ... We report the average of NDCG@5 scores of all the evaluated query contexts. (3) Recall@K, the proportion of query contexts whose gold quotes are ranked in respective top K candidate quotes, K={1,10,100}.

## Citation

```bibtex
@misc{qi2022quoter,
  title={QuoteR: A Benchmark of Quote Recommendation for Writing},
  author={Qi et al. (2022)},
  year={2022},
  note={arXiv:2202.13145}
}
```

- arXiv: 2202.13145