Multiclaim Eval

multiclaim-eval

Multilingual Previously Fact-Checked Claim Retrieval — Pikuliak et al. (2023) (arXiv:2305.07991, 2023)

What this evaluates

Tests the ability of retrieval models to rank previously fact-checked claims relevant to a given social media post. It probes crosslingual and monolingual claim retrieval capabilities, evaluating how well models handle multilingual text, varying post lengths, and different fact-check ratings.

Datasets

• MultiClaim — total 31000; splits: test (31000); repo https://github.com/kinit-sk/multiclaim

Metrics

• S@K (primary) — range: [0, 1]
  • Percentage of test pairs where the correct fact-check appears in the top K ranked results.

Input / output format

Input: Query: social media post (main text + OCR transcript). Candidate pool: fact-check claims (text only).

Output: Ranked list of fact-check claims sorted by relevance score.

Scoring recipe

def success_at_k(predictions, gold, k=10):
    hits = 0
    for gold_id in gold:
        if gold_id in predictions[:k]:
            hits += 1
    return hits / len(gold)

Common pitfalls

• Same language bias (SLB) can artificially inflate monolingual scores if the full multilingual fact-check pool is used without restricting to the post's language.
• BM25 performance is highly sensitive to script type (e.g., fails on Thai/Myanmar due to scriptio continua) and candidate pool size.
• Crosslingual evaluation requires searching the entire multilingual fact-check index, whereas monolingual evaluation restricts the search to the post's language pool.

Evidence (verbatim from paper)

We evaluate the performance based on the rank of the desired fact-checks by using success-at-K (S@K) as the main evaluation metric. We define as the percentage of pairs when the desired fact-check ends up in the top K.

Citation

@misc{pikuliak2023multiclaim,
  title={Multilingual Previously Fact-Checked Claim Retrieval},
  author={Pikuliak et al. (2023)},
  year={2023},
  note={arXiv:2305.07991}
}

• arXiv: 2305.07991