Evetar Eval

evetar-eval

EveTAR: Building a Large-Scale Multi-Task Test Collection over Arabic Tweets — Hasanain et al. (2017) (arXiv:1708.05517, 2017)

What this evaluates

EveTAR evaluates Arabic information retrieval systems across four tasks: event detection, ad-hoc search, tweet timeline generation, and real-time summarization. It probes a model's ability to retrieve, cluster, and summarize relevant Arabic tweets in response to event-driven queries or topics.

Datasets

• EveTAR — total 355000000; splits: EveTAR-F (-1), EveTAR-S (-1), EveTAR-S.m (-1), EveTAR-S.d (-1), EveTAR-Q (-1)

Metrics

• recall (primary) — range: [0, 1]
  • recall = #covered events / #reference events. A reference event is considered covered if at least 50% of the tweets in the detected event belong to that reference event.
• MAP — range: [0, 1]
  • Mean Average Precision averaged over all 50 topics.
• P@30 — range: [0, 1]
  • Precision at rank 30 averaged over all 50 topics.
• wF1 — range: [0, 1]
  • Weighted F1 score used as the official evaluation measure for tweet timeline generation in TREC 2014.
• EG-1 — range: [0, 1]
  • Expected Gain metric used for real-time summarization evaluation.
• nCG-1 — range: [0, 1]
  • Normalized Cumulative Gain metric used for real-time summarization evaluation.

Input / output format

Input: For each task, the system receives a collection of Arabic tweets. Event detection receives the full tweet collection. Ad-hoc search and TTG receive a query/topic and the tweet collection. RTS receives a topic title and a stream of tweets.

Output: Event detection: list of detected events (each a list of tweets). Ad-hoc search: ranked list of tweets. TTG: timeline of summarized tweets. RTS: set of pushed tweets or silent decision.

Scoring recipe

def compute_recall(detected_events, reference_events):
    covered = 0
    for ref_event in reference_events:
        ref_tweets = set(ref_event.tweets)
        for det_event in detected_events:
            det_tweets = set(det_event.tweets)
            if len(ref_tweets & det_tweets) >= 0.5 * len(ref_tweets):
                covered += 1
                break
    return covered / len(reference_events)

Common pitfalls

• Precision cannot be automatically computed for event detection because the complete set of possible significant events in the collection is unknown.
• Task performance is highly sensitive to language mismatch; e.g., RTS systems using Modern Standard Arabic topic titles on dialectal Arabic tweets yield artificially low precision.
• The Generalizability Coefficient (GC) measures ranking reliability for a specific set of systems, not the absolute quality of the test collection.

Evidence (verbatim from paper)

To automatically evaluate the algorithms, we adopted Petrovic's (Petrovič 2013) approach which measures effectiveness using recall: recall = #covered events / #reference events where # reference events is the number of events in EveTAR and # covered events is the number of reference events covered by the algorithm's detected events (where an event is represented by a list of tweets).

Citation

@misc{hasanain2017evetar,
  title={EveTAR: Building a Large-Scale Multi-Task Test Collection over Arabic Tweets},
  author={Hasanain et al. (2017)},
  year={2017},
  note={arXiv:1708.05517}
}

• arXiv: 1708.05517