Mfaq Eval

mfaq-eval

MFAQ: a Multilingual FAQ Dataset — De Bruyn et al. (2021) (arXiv:2109.12870, 2021)

What this evaluates

Evaluates multilingual and monolingual bi-encoder models on a FAQ retrieval task across 21 languages. It probes cross-lingual knowledge transfer, semantic robustness to lexical changes, and the impact of training data distribution on retrieval performance.

Datasets

• MFAQ — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/clips/mfaq

Metrics

• MRR (primary) — range: percent
  • Mean Reciprocal Rank: the average of 1/rank for the ground-truth answer across all queries, where rank is the position of the correct answer in the retrieved list.
• P@1 — range: percent
  • Precision at 1: 1 if the ground-truth answer is ranked first, 0 otherwise.
• R@5 — range: percent
  • Recall at 5: 1 if the ground-truth answer appears in the top 5 retrieved results, 0 otherwise.

Input / output format

Input: A question string (prepended with ) and a candidate answer string (prepended with ). The model encodes them into a shared embedding space for similarity-based retrieval.

Output: A ranked list of candidate answers for each query question, based on cosine similarity between question and answer embeddings.

Scoring recipe

def compute_metrics(retrieved_ranks, gold_idx):
    p_at_1 = 0.0
    mrr = 0.0
    r_at_5 = 0.0
    for ranks in retrieved_ranks:
        if gold_idx in ranks:
            rank = ranks.index(gold_idx) + 1
            mrr += 1.0 / rank
            if rank == 1: p_at_1 += 1.0
            if rank <= 5: r_at_5 += 1.0
    n = len(retrieved_ranks)
    return {'P@1': p_at_1 / n, 'MRR': mrr / n, 'R@5': r_at_5 / n}

Common pitfalls

• Batch size differences between models (e.g., DPR vs RoBERTa) can significantly impact reported performance, as noted in the text.
• Filtering training data to one page per domain flattens topic distribution and artificially inflates MRR compared to using the full unfiltered dataset.
• Cross-lingual evaluation translates queries to English while keeping answers in the original language, which tests translation robustness rather than pure multilingual retrieval.

Evidence (verbatim from paper)

In all our experiments, we use three metrics to evaluate the performance: precision-at-one $(\mathrm{P}@\mathrm{1})$ , mean reciprocal rank (MRR), and recall-at-5 $(\mathrm{R}@\mathrm{5})$ . For space reasons, we only report on MRR in the main text, the full results are available in the annex.

Citation

@misc{debruyn2021mfaq,
  title={MFAQ: a Multilingual FAQ Dataset},
  author={De Bruyn et al. (2021)},
  year={2021},
  note={arXiv:2109.12870}
}

• arXiv: 2109.12870