# Aratable Eval

> aratable-eval

- Skill: `qhjqhj00/aratable-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add qhjqhj00/aratable-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/aratable-eval/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/qhjqhj00/aratable-eval

---


# aratable-eval

> AraTable: Benchmarking LLMs' Reasoning and Understanding of Arabic Tabular Data — Alshaikh et al. (2025) (arXiv:2507.18442, 2025)

## What this evaluates

This benchmark probes large language models' ability to reason over and understand Arabic tabular data across three core tasks: direct question answering, fact verification, and complex reasoning. It specifically tests whether models can extract, compare, and synthesize information from structured Arabic tables while adhering to linguistic and formatting nuances.

## Datasets

- **AraTable** — total ?; splits: test (-1); repo https://github.com/rana-alshaikh/AraTable-Benchmark

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of model responses marked TRUE by human or LLM judges after applying a relaxed semantic rubric. Calculated as (number of TRUE evaluations) / (total number of evaluated instances).

## Input / output format

**Input**: Arabic questions derived from tabular data, ground-truth answers, and a detailed evaluation rubric specifying tolerance rules for wording, numbers, lists, and formatting.

**Output**: Free-form Arabic responses to the questions. No strict structural constraints are imposed on the model's generation, but outputs are evaluated against a relaxed semantic rubric.

## Scoring recipe

```python
def evaluate(prediction, ground_truth):
    pred = normalize_text(prediction)
    gt = normalize_text(ground_truth)
    if semantic_match(pred, gt): return True
    if is_boolean(gt):
        if any(p in pred for p in TRUE_PHRASES): return True
        if any(p in pred for p in FALSE_PHRASES): return False
    if is_numeric(gt):
        if abs(float(pred) - float(gt)) <= 0.005: return True
    if is_list(gt):
        if set(normalize_list(pred)) == set(normalize_list(gt)): return True
    if requires_arabic_entity(gt) and not is_arabic(pred): return False
    return False
accuracy = sum(evaluate(p, g) for p, g in zip(predictions, ground_truths)) / len(predictions)
```

## Common pitfalls

- Unconstrained free-form outputs lead to high verbosity and formatting inconsistencies, making exact string matching or embedding similarity unreliable for scoring.
- Models are highly sensitive to question numbering and listing order, sometimes missing questions or outputting unordered answers if formatting changes.
- Models may filter responses based on trigger words or switch to English mid-answer, indicating over-reliance on pre-training data rather than the provided table.

## Evidence (verbatim from paper)

> This design enabled (a) evaluation of the accuracy of the human-extracted response from Jais’ verbose output and comparison of human evaluation consistency when presented with the same answer in both concise (Model 4) and verbose (Model 5) forms – ideally, both should yield the same accuracy, as the core answer remains unchanged; and (b) the assessment of the robustness of our automated evaluation method in understanding and evaluating the LLMs’ responses that exhibited verbosity.

## Citation

```bibtex
@misc{alshaikh2025aratable,
  title={AraTable: Benchmarking LLMs' Reasoning and Understanding of Arabic Tabular Data},
  author={Alshaikh et al. (2025)},
  year={2025},
  note={arXiv:2507.18442}
}
```

- arXiv: 2507.18442