Trec Rts Metrics

trec-rts-metrics

Everything You Always Wanted to Know About TREC RTS* (*But Were Afraid to Ask) — Hubert et al. (2017) (arXiv:1712.04671, 2017)

What this evaluates

Evaluates real-time tweet summarization systems by measuring their ability to push relevant, non-redundant tweets within fixed temporal windows, while penalizing system latency and irrelevant outputs.

Datasets

• TREC RTS Track Data — total ?; splits: test (-1)

Metrics

• Expected Gain (EG) (primary) — range: [0, 1]
  • EG(w_j, S_i) = (1 / |T_i(w_j)|) * Σ g(t), where g(t)=1 if t is the first relevant tweet in its cluster, else 0. Variants handle silent days: EG-0 gives 0, EG-1 gives 1 if no tweets pushed, EG-p gives (N - non_rel)/N.
• Normalized Cumulative Gain (nCG) — range: [0, 1]
  • nCG(w_j, S_i) = (1 / Z) * G(S_i, w_j), where Z is the maximum possible gain given the N tweets/day limit. Shares EG-0/1/p silent day variants.
• Gain Minus Pain (GMP) — range: other
  • GMP = α * ΣG - (1-α) * P, where P is the count of non-relevant tweets returned. α is tuned to 0.33, 0.50, or 0.66.
• Latency — range: other
  • Sum over relevant tweets of (push_time - creation_time) for the oldest tweet pushed per cluster.

Input / output format

Input: For each temporal window and interest profile: a set of candidate tweets with creation timestamps, pre-defined semantic clusters, and the system's output list of pushed tweets with push timestamps.

Output: A sequence of pushed tweets per window, ordered by push time, with associated timestamps.

Scoring recipe

def compute_eg(pushed, window_tweets, N, variant="EG-1"):
    gain = 0
    seen_clusters = set()
    for t in window_tweets:
        if t.cluster not in seen_clusters:
            if t in pushed:
                gain += 1
                seen_clusters.add(t.cluster)
    n_pushed = len(pushed)
    if variant == "EG-0":
        return gain / n_pushed if n_pushed > 0 else 0
    elif variant == "EG-1":
        return 1.0 if n_pushed == 0 else gain / n_pushed
    else: # EG-p
        non_rel = n_pushed - gain
        return (N - non_rel) / N

Common pitfalls

• Relevance is system-dependent: a tweet is only relevant if it is the first one retrieved from its cluster by that specific system.
• Silent day handling variants (EG-0, EG-1, EG-p) drastically change scores and should not be mixed.
• Gain is calculated based on tweet publication time, not push time, so systems can score in windows where they returned nothing.
• The Latency metric can yield a perfect score without returning any relevant tweets.

Evidence (verbatim from paper)

The expected gain metric, denoted by EG, is adapted from [2]. Given a time window w_j, it is evaluated as: EG(w_j, S_i) = (1 / |T_i(w_j)|) * G(S_i, w_j) where |T_i(w_j)| is the number of tweets returned by S_i and published during w_j.

Citation

@misc{hubert2017trecrts,
  title={Everything You Always Wanted to Know About TREC RTS* (*But Were Afraid to Ask)},
  author={Hubert et al. (2017)},
  year={2017},
  note={arXiv:1712.04671}
}

• arXiv: 1712.04671