Qilin Eval

qilin-eval

Qilin: A Multimodal Information Retrieval Dataset with APP-level User Sessions — Jia Chen et al. (arXiv:2503.00501, 2025)

What this evaluates

Evaluates multimodal information retrieval systems across search, recommendation, and deep query answering (DQA) tasks using real-world APP-level user sessions. It probes a model's ability to rank heterogeneous content (text, images, videos) and generate accurate answers augmented by retrieved documents.

Datasets

• Qilin — total ?; splits: train (-1), test (-1); repo https://github.com/RED-Search/Qilin

Metrics

• MRR@10 (primary) — range: [0, 1]
  • Mean Reciprocal Rank at K: 1/N ∑(I(r_i ≤ K)/r_i), where N is the number of test instances and r_i is the rank of the first positive result.
• MRR@100 — range: [0, 1]
  • Same as MRR@10 but evaluated at K=100.
• MAP@10 — range: [0, 1]
  • Mean Average Precision at K: 1/N ∑(1/R_i ∑ P_i,k · I(rel_i,k=1)), where R_i is total relevant results, P_i,k is precision at rank k, and rel_i,k is binary relevance (clicks).
• MAP@100 — range: [0, 1]
  • Same as MAP@10 but evaluated at K=100.
• ROUGE-L — range: [0, 1]
  • Longest Common Subsequence-based F1 score measuring syntactic overlap between generated and reference answers.
• BERTScore (F1) — range: [0, 1]
  • Semantic similarity score computed using contextual embeddings from BERT, reporting the F1 harmonic mean of precision and recall.

Input / output format

Input: Query text, note title, note content, and optionally cover images. For recommendation, concatenated titles of recently clicked notes act as pseudo-query. For DQA, query plus retrieved reference documents (or none).

Output: Ranked list of notes (search/recommendation) or generated text answer (DQA).

Scoring recipe

def compute_mrr(ranked_rels, K):
    for rank, rel in enumerate(ranked_rels, 1):
        if rel == 1: return 1.0 / rank
    return 0.0
def compute_map(ranked_rels, K):
    prec, rel_cnt = [], 0
    for k, rel in enumerate(ranked_rels[:K], 1):
        if rel == 1:
            rel_cnt += 1
            prec.append(rel_cnt / k)
    return np.mean(prec) if prec else 0.0
mrr_k = np.mean([compute_mrr(r, K) for r in test_rels])
map_k = np.mean([compute_map(r, K) for r in test_rels])
rouge_l = rouge_l_score(pred, gold)
bertscore_f1 = bertscore_score(pred, gold, lang='auto')

Common pitfalls

• Time-based split (11:1 ratio by hour) may cause data leakage if future sessions influence past training.
• Using clicks as binary relevance labels ignores implicit negative feedback and may introduce noise.
• DQA evaluation uses zero-shot RAG without fine-tuning, which may not reflect real-world deployment performance.

Evidence (verbatim from paper)

For evaluation metrics, we choose Mean Reciprocal Rank (MRR) and Mean Average Precision (MAP), which can be formulated as: MRR@K = 1/N ∑(I(r_i ≤ K)/r_i), MAP@K = 1/N ∑(1/R_i ∑ P_i,k · I(rel_i,k=1)) where N denotes the total number of testing instances, r_i represents the rank of the first positive result in the reranked list. As for rel_i,k, we use clicks as binary relevance labels for simplicity.

Citation

@misc{chen2025qilin,
  title={Qilin: A Multimodal Information Retrieval Dataset with APP-level User Sessions},
  author={Jia Chen et al.},
  year={2025},
  note={arXiv:2503.00501}
}

• arXiv: 2503.00501