qhjqhj00 ciqi-bench-eval
CiQi-Agent: Aligning Vision, Tools and Aesthetics in Multimodal Agent for Cultural Reasoning on Chinese Porcelains — Wang et al. (2026) (arXiv:2603.28474, 2026)
What this evaluates
Evaluates a multimodal agent's ability to perform fine-grained visual classification and cultural reasoning on antique Chinese porcelain. It probes seven specific connoisseurship attributes (dynasty, reign period, kiln site, glaze color, decorative motif, vessel shape, and overall naming) through both multiple-choice and free-form generation tasks.
Datasets
- CiQi-Bench — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Percentage of correctly answered questions. For multiple-choice tasks, it is the fraction of instances where the model's selected option matches the ground truth. For free-form tasks, correctness is determined by an LLM-as-a-Judge evaluator (Qwen2.5-72B-Instruct).
Input / output format
Input: An image of a Chinese porcelain artifact paired with a question (multiple-choice or free-form) asking for connoisseurship attributes.
Output: For multiple-choice: a single letter/option corresponding to the correct attribute. For free-form: a generated text string describing the attribute.
Scoring recipe
def compute_accuracy(predictions, golds, use_llm_judge=False):
correct = 0
for pred, gold in zip(predictions, golds):
if use_llm_judge:
score = llm_judge.evaluate(pred, gold)
if score == 1: correct += 1
else:
if pred == gold: correct += 1
return (correct / len(golds)) * 100
Common pitfalls
- Free-form answers are evaluated by an LLM-as-a-Judge rather than exact string matching, which may introduce evaluator bias.
- Baselines are evaluated with temperature=0.0 for deterministic outputs, so stochastic decoding baselines are not directly comparable.
- The retrieval database is strictly non-overlapping with the test set to prevent information leakage, which must be replicated for fair evaluation.
Evidence (verbatim from paper)
We report accuracy as the primary metric for multiple-choice questions, computed as the percentage of correctly answered questions across all seven dimensions: overall naming, dynasty, reign period, kiln site, glaze color, decorative motif, and vessel shape. For free-form generation tasks, we employ Qwen2.5-72B-Instruct as the LLM-as-a-Judge evaluator.
Citation
@misc{wang2026ciqiagent,
title={CiQi-Agent: Aligning Vision, Tools and Aesthetics in Multimodal Agent for Cultural Reasoning on Chinese Porcelains},
author={Wang et al. (2026)},
year={2026},
note={arXiv:2603.28474}
}
- arXiv: 2603.28474