# Neucir 2023 Eval

> Evaluates neural cross-language and multilingual information retrieval systems on news collections in Chinese, Persian, and Russian. It probes a model's ability to rank relevant documents when queries are in English and documents are in different languages, as well as its capacity to unify rankings across multiple languages. Use when the user wants to benchmark on NeuCLIR 2023, or asks about evaluating this task. Reports nDCG@20.

- Skill: `qhjqhj00/neucir-2023-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/neucir-2023-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/neucir-2023-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/neucir-2023-eval

---


# neucir-2023-eval

> Overview of the TREC 2023 NeuCLIR Track — Dawn Lawrie et al. (arXiv:2404.08071, 2024)

## What this evaluates

Evaluates neural cross-language and multilingual information retrieval systems on news collections in Chinese, Persian, and Russian. It probes a model's ability to rank relevant documents when queries are in English and documents are in different languages, as well as its capacity to unify rankings across multiple languages.

## Datasets

- **NeuCLIR 2023** — total ?; splits: test (-1)

## Metrics

- `nDCG@20` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 20. Computes the sum of graded relevance values of results up to position 20, discounted logarithmically by rank, and normalizes by the ideal DCG@20.

## Input / output format

**Input**: English query (topic) and a document collection in Chinese, Persian, or Russian. For reranking, an initial ranked list of 1,000 documents is also provided.

**Output**: A ranked list of 1,000 document IDs. For the MLIR task, a single unified ranked list across all three language collections.

## Scoring recipe

```python
def compute_ndcg_at_20(relevance_scores, k=20):
    dcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(relevance_scores[:k]))
    ideal = sorted(relevance_scores, reverse=True)[:k]
    idcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(ideal))
    return dcg / idcg if idcg > 0 else 0.0
```

## Common pitfalls

- Ad-hoc CLIR requires searching the entire collection, while reranking CLIR only reorders a pre-provided list of 1,000 documents.
- MLIR requires producing a single unified ranked list across Chinese, Persian, and Russian collections, not separate per-language lists.
- Relevance judgments use a 4-point scale that is converted to a 3-point scale for qrels; failing to apply this conversion will skew metric values.

## Evidence (verbatim from paper)

> The CLIR runs summarized in Figure 1 substantially outperformed the monolingual runs this year; this is different from last year, when the top runs from CLIR and monolingual tasks had similar nDCG@20.

## Citation

```bibtex
@misc{lawrie2024neucir,
  title={Overview of the TREC 2023 NeuCLIR Track},
  author={Dawn Lawrie et al.},
  year={2024},
  note={arXiv:2404.08071}
}
```

- arXiv: 2404.08071