qhjqhj00 cirthan-eval
A Sketch+Text Composed Image Retrieval Dataset for Thangka — Xu et al. (2026) (arXiv:2602.08411, 2026)
What this evaluates
Evaluates composed image retrieval capabilities on culturally specific Thangka imagery. It tests the model's ability to align fine-grained sketch+text queries with target images across varying levels of textual semantic granularity, highlighting the domain gap between generic pre-training and specialized cultural retrieval.
Datasets
- CIRThan — total 2287; splits: train (1868), test (419); repo https://github.com/jinyuxu-whut/CIRThan
Metrics
Recall@K (R@K)(primary) — range: percent- For each composed query, all gallery images are ranked by similarity. Retrieval is considered successful if the ground-truth target image appears within the top-K results. Reported for K ∈ {1, 3, 5} and averaged across all queries.
Input / output format
Input: A composed query consisting of a human-drawn sketch paired with a hierarchical textual description (Level 1, 2, or 3), and a gallery of candidate Thangka images.
Output: A ranked list of gallery images by similarity score, or a binary success indicator per query for top-K evaluation.
Scoring recipe
def compute_recall_at_k(predictions, gold_indices, k):
hits = 0
for pred, gold in zip(predictions, gold_indices):
if gold in pred[:k]:
hits += 1
return hits / len(predictions) * 100
Common pitfalls
- Zero-shot methods struggle significantly with domain-specific visual semantics, often failing to align sketch and text cues without in-domain supervision.
- Single-modality baselines (sketch-only or text-only) perform poorly, highlighting the necessity of cross-modal fine-grained alignment for Thangka retrieval.
- Retrieval performance is highly sensitive to textual granularity; Level 3 descriptions consistently yield higher recall than Level 1 or 2.
Evidence (verbatim from paper)
Following common practice in composed image retrieval, we evaluate all methods using Recall@K (R@K). For each composed query, all Thangka images in the gallery are ranked by similarity, and retrieval is considered successful if the ground-truth target Thangka appears within the top-K results. We report R@K for $K\in{1,3,5}$, as each query in CIRThan has a single ground-truth target.
Citation
@misc{xu2026cirthan,
title={A Sketch+Text Composed Image Retrieval Dataset for Thangka},
author={Xu et al. (2026)},
year={2026},
note={arXiv:2602.08411}
}
- arXiv: 2602.08411