Contextual Ir Eval

contextual-ir-eval

Using Context to Improve the Evaluation of Information Retrieval Systems — Bouramoul et al. (2011) (arXiv:1105.6213, 2011)

What this evaluates

Evaluates information retrieval systems by measuring system-level performance metrics (dead links, response time, redundancy) and user-perceived relevance across different query topics and rank positions.

Datasets

• Custom IR Evaluation Corpus — total 1800; splits: test (1800)

Metrics

• Dead Link Rate — range: percent
  • Percentage of retrieved URLs that fail to load or return a 404 error after up to three retry attempts.
• Parasite Page Rate — range: percent
  • Percentage of results linking to commercial sites offering online purchases or transactions.
• Redundant Result Rate — range: percent
  • Percentage of results containing duplicate or alias links to the same content.
• Average Response Time — range: other
  • Mean time in seconds from query submission to result display, measured on a single machine with fixed internet speed.
• Relevance Judgments (primary) — range: other
  • Average rating on a 0-5 scale assigned by non-expert users to the top-k results (k=1,5,10,15,20), where 0 is completely useless/off-topic and 5 is perfect.

Input / output format

Input: Query string, search engine identifier, and retrieved URL/page content (organized as triplets).

Output: System metrics: boolean flags for dead links, parasite pages, redundancy, and float for response time. Relevance metrics: integer score 0-5 per result, or float score 0-10 per topic group.

Scoring recipe

def compute_metrics(urls, response_times, user_ratings, queries, results):
    dead_links = sum(1 for u in urls if u.is_dead) / len(urls) * 100
    parasites = sum(1 for u in urls if u.is_commercial) / len(urls) * 100
    redundant = sum(1 for u in urls if u.is_duplicate) / len(urls) * 100
    avg_time = sum(response_times) / len(response_times)
    
    relevance_scores = [r for r in user_ratings if 0 <= r <= 5]
    avg_relevance = sum(relevance_scores) / len(relevance_scores)
    
    topic_scores = [count_query_terms_in_result(q, r) for q, r in zip(queries, results)]
    avg_topic_relevance = sum(topic_scores) / len(topic_scores) * 2
    
    return dead_links, parasites, redundant, avg_time, avg_relevance, avg_topic_relevance

Common pitfalls

• Response time measurements are highly dependent on the specific hardware and network conditions used during the experiment, making cross-study comparisons difficult.
• The custom relevance formula for query-topic scoring relies on word occurrence counts, which may underperform for short queries lacking sufficient terms.
• User judgments come from non-expert students, which may not generalize to professional relevance raters or real-world user behavior.

Evidence (verbatim from paper)

The 24 students also expressed their relevance judgments for 5, 10, 15 and 20 first retrieved documents (R@5, R@10, R@10, R@15, R@20). At each level of relevance, a note of 0-5 was assigned by each student. 0 corresponding to a document completely useless or off-topic, 5 corresponding to a document responding in a perfect way to the question.

Citation

@misc{bouramoul2011using,
  title={Using Context to Improve the Evaluation of Information Retrieval Systems},
  author={Bouramoul et al. (2011)},
  year={2011},
  note={arXiv:1105.6213}
}

• arXiv: 1105.6213