Danqing Eval

danqing-eval

DanQing: An Up-to-Date Large-Scale Chinese Vision-Language Pre-training Dataset — Shen et al. (2026) (arXiv:2601.10305, 2026)

What this evaluates

Evaluates the quality of Chinese vision-language pre-training datasets by measuring downstream performance on cross-modal retrieval and multimodal reasoning benchmarks after continual pre-training.

Datasets

• Flickr30K-CN — total ?; splits: test (-1)
• MSCOCO-CN — total ?; splits: test (-1)
• MUGE — total ?; splits: test (-1)
• DCI-CN — total ?; splits: test (-1)
• DOCCI-CN — total ?; splits: test (-1)

Metrics

• R@1/5/10 (primary) — range: percent
  • Recall@K measures the fraction of relevant items retrieved within the top K results. Computed separately for text-to-image and image-to-text directions.

Input / output format

Input: Image and corresponding text caption for retrieval; image and question for LMM benchmarks.

Output: Retrieval ranking scores (R@1, R@5, R@10) per direction; accuracy scores for LMM benchmarks.

Scoring recipe

def compute_recall_at_k(retrieved_top_k, relevant_set, k):
    hits = sum(1 for idx in retrieved_top_k[:k] if idx in relevant_set)
    return hits / len(relevant_set)

def evaluate_retrieval(model, test_pairs):
    metrics = {'R@1': 0.0, 'R@5': 0.0, 'R@10': 0.0}
    for query, gold_idx in test_pairs:
        scores = model.encode(query)
        top_k = get_top_k_indices(scores, k=10)
        for k in [1, 5, 10]:
            metrics[f'R@{k}'] += compute_recall_at_k(top_k, {gold_idx}, k)
    for k in [1, 5, 10]:
        metrics[f'R@{k}'] /= len(test_pairs)
    return metrics

Common pitfalls

• Competitor datasets (Zero, TaiSu) are randomly sampled to 100M pairs to match DanQing's size, which may introduce sampling variance and affect reproducibility.
• Long-caption retrieval is constrained to a 64-token context length, potentially truncating relevant information and lowering recall scores.
• Metrics are reported separately for text-to-image and image-to-text directions, but averages are sometimes combined without clear weighting or direction specification.

Evidence (verbatim from paper)

Table 3: Cross-modal retrieval performance on short-caption datasets for models pretrained on various large-scale Chinese image-text datasets. * indicates random sampling of 100 million image-text pairs. The best and second-best results are highlighted in bold and underlined, respectively. R@1 R@5 R@10

Citation

@misc{shen2026danqing,
  title={DanQing: An Up-to-Date Large-Scale Chinese Vision-Language Pre-training Dataset},
  author={Shen et al. (2026)},
  year={2026},
  note={arXiv:2601.10305}
}

• arXiv: 2601.10305