# Arabic Evidence Retrieval Eval

> This benchmark tests a system's ability to retrieve relevant evidence snippets from a large pool of web pages for a given Arabic claim. It evaluates ranking performance in a retrieval setting where only a small fraction of snippets actually contain verifying evidence. Use when the user wants to benchmark on Arabic Evidence Retrieval Dataset, or asks about evaluating this task. Reports P@10.

- Skill: `qhjqhj00/arabic-evidence-retrieval-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/arabic-evidence-retrieval-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/arabic-evidence-retrieval-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/arabic-evidence-retrieval-eval

---


# arabic-evidence-retrieval-eval

> Overview of CheckThat! 2020: Automatic Identification and Verification of Claims in Social Media — Barrón-Cedeno et al. (2020) (arXiv:2007.07997, 2020)

## What this evaluates

This benchmark tests a system's ability to retrieve relevant evidence snippets from a large pool of web pages for a given Arabic claim. It evaluates ranking performance in a retrieval setting where only a small fraction of snippets actually contain verifying evidence.

## Datasets

- **Arabic Evidence Retrieval Dataset** — total 169902; splits: test (169902)

## Metrics

- `P@10` **(primary)** — range: [0, 1]
  - Precision at cutoff k=10, measuring the proportion of retrieved snippets that contain useful evidence among the top 10 results.

## Input / output format

**Input**: Arabic claim text and a list of candidate web page snippets.

**Output**: Ranked list of evidence snippets.

## Scoring recipe

```python
def score(predictions, gold):
    top_k = predictions[:10]
    relevant_in_top_k = len(set(top_k) & gold)
    return relevant_in_top_k / 10.0
```

## Common pitfalls

- Evaluation relies on a pooling method: only snippets returned by systems are annotated, which may bias precision estimates.
- Snippets are automatically split by punctuation, which might not align with semantic evidence boundaries.

## Evidence (verbatim from paper)

> In order to evaluate the submitted runs, we computed P@k at different cutoff (k = 1, 5, 10). The official measure was P@10.

## Citation

```bibtex
@misc{barroncedeno2020checkthat,
  title={Overview of CheckThat! 2020: Automatic Identification and Verification of Claims in Social Media},
  author={Barrón-Cedeno et al. (2020)},
  year={2020},
  note={arXiv:2007.07997}
}
```

- arXiv: 2007.07997