orca-eval
ORCA: A Challenging Benchmark for Arabic Language Understanding — Elmadany et al. (2022) (arXiv:2212.10758, 2022)
What this evaluates
Evaluates Arabic language understanding across seven task clusters, including sentence classification, structured prediction, semantic similarity, NLI, QA, WSD, and topic classification. It probes models' ability to handle diverse Arabic varieties (MSA and dialects) and multiple linguistic levels from tokens to documents.
Datasets
- ORCA — total 588200; splits: train (487100), dev (46000), test (55100)
Metrics
ORCA score (primary) — range: [0, 1]
- A macro-average of the performance scores across all 29 tasks and 7 task clusters, with each task weighted equally. Per-task scores are computed using standard metrics for each task type (e.g., accuracy, F1, Pearson correlation).
Input / output format
Input: Varies by task cluster: single sentences for classification, sentence pairs for STS/NLI, token sequences for NER/POS, and question-context pairs for QA.
Output: Task-specific predictions: class labels, span boundaries, similarity scores, or generated answers.
Scoring recipe
task_scores = []
for task in all_29_tasks:
preds, golds = get_predictions_and_labels(task)
task_scores.append(compute_task_metric(preds, golds))
orca_score = sum(task_scores) / len(task_scores)
Common pitfalls
- Dataset sizes are capped at 50k/5k/5k for GPU-friendly evaluation, which may not reflect performance on original full-scale datasets.
- The ORCA score treats all 29 tasks equally, potentially masking performance differences across task clusters or linguistic varieties.
- Leaderboard submissions require strict metadata (parameter count, pretraining data size, fine-tuning epochs) for fair comparison.
Evidence (verbatim from paper)
Simple evaluation metric. We adopt a simple evaluation approach in the form of an ORCA score. The ORCA score is simply a macro-average of the different scores across all tasks and task clusters, where each task is weighted equally.
Citation
@misc{elmadany2022orca,
title={ORCA: A Challenging Benchmark for Arabic Language Understanding},
author={Elmadany et al. (2022)},
year={2022},
note={arXiv:2212.10758}
}
1---2name: orca-eval3description: Evaluates Arabic language understanding across seven task clusters, including sentence classification, structured prediction, semantic similarity, NLI, QA, WSD, and topic classification. It probes models' ability to handle diverse Arabic varieties (MSA and dialects) and multiple linguistic levels from tokens to documents. Use when the user wants to benchmark on ORCA, or asks about evaluating this task. Reports ORCA score.4---56# orca-eval78> ORCA: A Challenging Benchmark for Arabic Language Understanding — Elmadany et al. (2022) (arXiv:2212.10758, 2022)910## What this evaluates1112Evaluates Arabic language understanding across seven task clusters, including sentence classification, structured prediction, semantic similarity, NLI, QA, WSD, and topic classification. It probes models' ability to handle diverse Arabic varieties (MSA and dialects) and multiple linguistic levels from tokens to documents.1314## Datasets1516- **ORCA** — total 588200; splits: train (487100), dev (46000), test (55100)1718## Metrics1920- `ORCA score` **(primary)** — range: [0, 1]21 - A macro-average of the performance scores across all 29 tasks and 7 task clusters, with each task weighted equally. Per-task scores are computed using standard metrics for each task type (e.g., accuracy, F1, Pearson correlation).2223## Input / output format2425**Input**: Varies by task cluster: single sentences for classification, sentence pairs for STS/NLI, token sequences for NER/POS, and question-context pairs for QA.2627**Output**: Task-specific predictions: class labels, span boundaries, similarity scores, or generated answers.2829## Scoring recipe3031```python32task_scores = []33for task in all_29_tasks:34 preds, golds = get_predictions_and_labels(task)35 task_scores.append(compute_task_metric(preds, golds))36orca_score = sum(task_scores) / len(task_scores)37```3839## Common pitfalls4041- Dataset sizes are capped at 50k/5k/5k for GPU-friendly evaluation, which may not reflect performance on original full-scale datasets.42- The ORCA score treats all 29 tasks equally, potentially masking performance differences across task clusters or linguistic varieties.43- Leaderboard submissions require strict metadata (parameter count, pretraining data size, fine-tuning epochs) for fair comparison.4445## Evidence (verbatim from paper)4647> Simple evaluation metric. We adopt a simple evaluation approach in the form of an ORCA score. The ORCA score is simply a macro-average of the different scores across all tasks and task clusters, where each task is weighted equally.4849## Citation5051```bibtex52@misc{elmadany2022orca,53 title={ORCA: A Challenging Benchmark for Arabic Language Understanding},54 author={Elmadany et al. (2022)},55 year={2022},56 note={arXiv:2212.10758}57}58```5960- arXiv: 2212.10758
qhjqhj00