# Evetar Eval

> evetar-eval

- Skill: `qhjqhj00/evetar-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add qhjqhj00/evetar-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/evetar-eval/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/qhjqhj00/evetar-eval

---


# evetar-eval

> EveTAR: Building a Large-Scale Multi-Task Test Collection over Arabic Tweets — Hasanain et al. (2017) (arXiv:1708.05517, 2017)

## What this evaluates

EveTAR evaluates Arabic information retrieval systems across four tasks: event detection, ad-hoc search, tweet timeline generation, and real-time summarization. It probes a model's ability to retrieve, cluster, and summarize relevant Arabic tweets in response to event-driven queries or topics.

## Datasets

- **EveTAR** — total 355000000; splits: EveTAR-F (-1), EveTAR-S (-1), EveTAR-S.m (-1), EveTAR-S.d (-1), EveTAR-Q (-1)

## Metrics

- `recall` **(primary)** — range: [0, 1]
  - recall = #covered events / #reference events. A reference event is considered covered if at least 50% of the tweets in the detected event belong to that reference event.
- `MAP` — range: [0, 1]
  - Mean Average Precision averaged over all 50 topics.
- `P@30` — range: [0, 1]
  - Precision at rank 30 averaged over all 50 topics.
- `wF1` — range: [0, 1]
  - Weighted F1 score used as the official evaluation measure for tweet timeline generation in TREC 2014.
- `EG-1` — range: [0, 1]
  - Expected Gain metric used for real-time summarization evaluation.
- `nCG-1` — range: [0, 1]
  - Normalized Cumulative Gain metric used for real-time summarization evaluation.

## Input / output format

**Input**: For each task, the system receives a collection of Arabic tweets. Event detection receives the full tweet collection. Ad-hoc search and TTG receive a query/topic and the tweet collection. RTS receives a topic title and a stream of tweets.

**Output**: Event detection: list of detected events (each a list of tweets). Ad-hoc search: ranked list of tweets. TTG: timeline of summarized tweets. RTS: set of pushed tweets or silent decision.

## Scoring recipe

```python
def compute_recall(detected_events, reference_events):
    covered = 0
    for ref_event in reference_events:
        ref_tweets = set(ref_event.tweets)
        for det_event in detected_events:
            det_tweets = set(det_event.tweets)
            if len(ref_tweets & det_tweets) >= 0.5 * len(ref_tweets):
                covered += 1
                break
    return covered / len(reference_events)
```

## Common pitfalls

- Precision cannot be automatically computed for event detection because the complete set of possible significant events in the collection is unknown.
- Task performance is highly sensitive to language mismatch; e.g., RTS systems using Modern Standard Arabic topic titles on dialectal Arabic tweets yield artificially low precision.
- The Generalizability Coefficient (GC) measures ranking reliability for a specific set of systems, not the absolute quality of the test collection.

## Evidence (verbatim from paper)

> To automatically evaluate the algorithms, we adopted Petrovic's (Petrovič 2013) approach which measures effectiveness using recall: recall = #covered events / #reference events where # reference events is the number of events in EveTAR and # covered events is the number of reference events covered by the algorithm's detected events (where an event is represented by a list of tweets).

## Citation

```bibtex
@misc{hasanain2017evetar,
  title={EveTAR: Building a Large-Scale Multi-Task Test Collection over Arabic Tweets},
  author={Hasanain et al. (2017)},
  year={2017},
  note={arXiv:1708.05517}
}
```

- arXiv: 1708.05517