Xtc Bench Eval

xtc-bench-eval

Beyond Accuracy: Benchmarking Cross-Task Consistency in Unified Multimodal Models — Wang et al. (2026) (arXiv:2604.25072, 2026)

What this evaluates

Evaluates cross-task semantic consistency in unified multimodal models by measuring how well generation and understanding tasks align on shared scene-graph facts. It probes whether architectural unification leads to representation-level coherence or merely independent task accuracy, specifically highlighting failures like consistent hallucination.

Datasets

• XTC-Bench — total ?; splits: test (-1)

Metrics

• CCTA (primary) — range: [0, 1]
  • Continuous Cross-Task Agreement (CCTA) measures the proportion of shared facts (objects, attributes, relations) between the generated image and the understood scene graph. It quantifies raw semantic alignment regardless of factual correctness.
• AW-CCTA (primary) — range: [0, 1]
  • Accuracy-Weighted CCTA (AW-CCTA) weights the raw CCTA score by the factual accuracy of both generation and understanding tasks. It penalizes consistent hallucination where both tasks agree but are incorrect.
• Generation score (G) — range: [0, 1]
  • Per-task performance evaluated on scene graph nodes, reporting OverallGen, MatchedNodes, and accuracy for attributes and relations.
• Understanding score (U) — range: [0, 1]
  • Per-task performance measured via scene graph-derived VQA, reporting OverallUnd, object retrieval, attribute queries, and relation queries.

Input / output format

Input: Scene graph descriptions used to generate prompts for image generation and VQA queries for understanding.

Output: Generated images and textual answers to VQA queries.

Scoring recipe

def compute_ccta(gen_facts, understand_facts):
    shared = set(gen_facts) & set(understand_facts)
    return len(shared) / max(len(gen_facts), 1)

def compute_aw_ccta(gen_facts, understand_facts, gold_facts):
    gen_acc = sum(1 for f in gen_facts if f in gold_facts) / len(gen_facts)
    und_acc = sum(1 for f in understand_facts if f in gold_facts) / len(understand_facts)
    raw_ccta = compute_ccta(gen_facts, understand_facts)
    return raw_ccta * gen_acc * und_acc

Common pitfalls

• Raw CCTA rewards consistent hallucination equally to consistent correctness, masking factual errors.
• Per-task accuracy (G and U) does not predict cross-task consistency, so evaluating them alone misses semantic misalignment.
• Matched-node evaluation converges for generation but not understanding, indicating relational comprehension is a persistent bottleneck.

Evidence (verbatim from paper)

We evaluate all models along three complementary dimensions: generation (G), understanding(U), and cross-task consistency (CCTA and AW-CCTA). First, cross-task inconsistency is observed universally: the strongest CCTA reaches only 0.706, and under AW-CCTA the best model achieves 0.623, demonstrating that semantic misalignment is a structural property not resolved by scale or architectural sophistication.

Citation

@misc{wang2026xtcbench,
  title={Beyond Accuracy: Benchmarking Cross-Task Consistency in Unified Multimodal Models},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2604.25072}
}

• arXiv: 2604.25072