# Contextual Ir Eval

> Evaluates information retrieval systems by measuring system-level performance metrics (dead links, response time, redundancy) and user-perceived relevance across different query topics and rank positions. Use when the user wants to benchmark on Custom IR Evaluation Corpus, or asks about evaluating this task. Reports Relevance Judgments.

- Skill: `qhjqhj00/contextual-ir-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/contextual-ir-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/contextual-ir-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/contextual-ir-eval

---


# contextual-ir-eval

> Using Context to Improve the Evaluation of Information Retrieval Systems — Bouramoul et al. (2011) (arXiv:1105.6213, 2011)

## What this evaluates

Evaluates information retrieval systems by measuring system-level performance metrics (dead links, response time, redundancy) and user-perceived relevance across different query topics and rank positions.

## Datasets

- **Custom IR Evaluation Corpus** — total 1800; splits: test (1800)

## Metrics

- `Dead Link Rate` — range: percent
  - Percentage of retrieved URLs that fail to load or return a 404 error after up to three retry attempts.
- `Parasite Page Rate` — range: percent
  - Percentage of results linking to commercial sites offering online purchases or transactions.
- `Redundant Result Rate` — range: percent
  - Percentage of results containing duplicate or alias links to the same content.
- `Average Response Time` — range: other
  - Mean time in seconds from query submission to result display, measured on a single machine with fixed internet speed.
- `Relevance Judgments` **(primary)** — range: other
  - Average rating on a 0-5 scale assigned by non-expert users to the top-k results (k=1,5,10,15,20), where 0 is completely useless/off-topic and 5 is perfect.

## Input / output format

**Input**: Query string, search engine identifier, and retrieved URL/page content (organized as triplets).

**Output**: System metrics: boolean flags for dead links, parasite pages, redundancy, and float for response time. Relevance metrics: integer score 0-5 per result, or float score 0-10 per topic group.

## Scoring recipe

```python
def compute_metrics(urls, response_times, user_ratings, queries, results):
    dead_links = sum(1 for u in urls if u.is_dead) / len(urls) * 100
    parasites = sum(1 for u in urls if u.is_commercial) / len(urls) * 100
    redundant = sum(1 for u in urls if u.is_duplicate) / len(urls) * 100
    avg_time = sum(response_times) / len(response_times)
    
    relevance_scores = [r for r in user_ratings if 0 <= r <= 5]
    avg_relevance = sum(relevance_scores) / len(relevance_scores)
    
    topic_scores = [count_query_terms_in_result(q, r) for q, r in zip(queries, results)]
    avg_topic_relevance = sum(topic_scores) / len(topic_scores) * 2
    
    return dead_links, parasites, redundant, avg_time, avg_relevance, avg_topic_relevance
```

## Common pitfalls

- Response time measurements are highly dependent on the specific hardware and network conditions used during the experiment, making cross-study comparisons difficult.
- The custom relevance formula for query-topic scoring relies on word occurrence counts, which may underperform for short queries lacking sufficient terms.
- User judgments come from non-expert students, which may not generalize to professional relevance raters or real-world user behavior.

## Evidence (verbatim from paper)

> The 24 students also expressed their relevance judgments for 5, 10, 15 and 20 first retrieved documents (R@5, R@10, R@10, R@15, R@20). At each level of relevance, a note of 0-5 was assigned by each student. 0 corresponding to a document completely useless or off-topic, 5 corresponding to a document responding in a perfect way to the question.

## Citation

```bibtex
@misc{bouramoul2011using,
  title={Using Context to Improve the Evaluation of Information Retrieval Systems},
  author={Bouramoul et al. (2011)},
  year={2011},
  note={arXiv:1105.6213}
}
```

- arXiv: 1105.6213