Trec Microblog Eval

trec-microblog-eval

Knowledge-based Query Expansion in Real-Time Microblog Search — Qiang et al. (2015) (arXiv:1503.03961, 2015)

What this evaluates

Evaluates the effectiveness of query expansion methods for real-time microblog search by measuring how well ranked document lists match relevance judgments for short-form social media queries. It specifically probes the model's ability to handle vocabulary mismatch and temporal relevance in noisy, short-text retrieval scenarios.

Datasets

• TREC Microblog Track — total ?; splits: tuning (-1), test (-1)

Metrics

• MAP (primary) — range: [0, 1]
  • Mean Average Precision: the average of precision values computed at each rank where a relevant document is retrieved, averaged across all queries.
• P@30 — range: [0, 1]
  • Precision at rank 30: the proportion of relevant documents found within the top 30 retrieved results for each query, averaged across all queries.

Input / output format

Input: Query topics (short text strings) from the TREC Microblog track.

Output: Ranked list of microblog documents retrieved for each query.

Scoring recipe

def compute_metrics(predictions, gold):
    aps, p30s = [], []
    for qid, ranked_docs in predictions.items():
        relevant = gold[qid]
        hits = 0
        for i, doc in enumerate(ranked_docs):
            if doc in relevant:
                hits += 1
                aps.append(hits / (i + 1))
        p30s.append(hits / min(30, len(ranked_docs)))
    return {'MAP': sum(aps) / len(aps), 'P@30': sum(p30s) / len(p30s)}

Common pitfalls

• TREC'12 evaluation uses the highrel relevance set, while TREC'13 uses the allrel set; mixing these relevance definitions invalidates cross-year comparisons.
• Statistical significance must be verified using a paired t-test (p < 0.05) rather than relying solely on reported percentage improvements over baselines.

Evidence (verbatim from paper)

Table 4 shows the performance comparison of different query expansion methods. For statistical significance, we used a paired t-test. †, ‡, ‡ and § indicate that the corresponding improvements over SimpleKL, QESMM, QEWiki and RTRM are statistically significant ( $p < 0.05$ ), respectively. Note that all the methods listed in the table estimate the document model as SimpleKL. As we can see, all of the query expansion methods have significant MAP and P@30 improvements compared with the SimpleKL method, which indicates the effectiveness of query expansion in microblog retrieval. ... Note that for TREC'12, the ranking scores are computed with respect to the highrel set (Soboroff, Ounis, and Lin 2013); while for TREC'13, the scores are computed in the allrel set (Lin and Efron 2014).

Citation

@misc{qiang2015knowledgebased,
  title={Knowledge-based Query Expansion in Real-Time Microblog Search},
  author={Qiang et al. (2015)},
  year={2015},
  note={arXiv:1503.03961}
}

• arXiv: 1503.03961