Ccfqa Eval

ccfqa-eval

CCFQA: A Benchmark for Cross-Lingual and Cross-Modal Speech and Text Factuality Evaluation — Du et al. (2025) (arXiv:2508.07295, 2025)

What this evaluates

This benchmark evaluates the factual accuracy and consistency of multimodal large language models (MLLMs) when answering questions in text or speech modalities across eight languages. It specifically probes cross-lingual transfer capabilities and cross-modal alignment by measuring how well models maintain factual correctness when switching between languages or between text and audio inputs.

Datasets

• CCFQA — total 14400; splits: test (14400); repo https://github.com/yxduir/ccfqa

Metrics

• F1 score (primary) — range: [0, 100]
  • Token-level F1 score calculated between the model's generated answer and the ground truth reference answer. It measures the harmonic mean of precision and recall over token sets, penalizing both over-generation and under-generation.
• LLM-based accuracy — range: [0, 100]
  • Binary accuracy score determined by an LLM judge that compares the model's output against the ground truth. The evaluation prompt explicitly requires answers without explanations to isolate factual correctness from verbosity.
• Consistency — range: [0, 100]
  • Ratio of performance across cross-lingual (XQA/XSQA) and cross-modal (QA/SQA) task pairs, measuring how much performance degrades when changing language or modality.

Input / output format

Input: Text questions or audio recordings of questions in 8 languages (Mandarin, English, French, Japanese, Korean, Russian, Spanish, Cantonese). Models receive either plain text or speech audio as input.

Output: Factual answers to the questions. The evaluation prompt explicitly requires answers without explanations to ensure clean token matching for F1 calculation.

Scoring recipe

def compute_metrics(preds, golds):
    f1s, accs = [], []
    for p, g in zip(preds, golds):
        p_tok, g_tok = set(p.lower().split()), set(g.lower().split())
        if not p_tok and not g_tok: f1s.append(1.0)
        elif not p_tok or not g_tok: f1s.append(0.0)
        else:
            prec = len(p_tok & g_tok) / len(p_tok)
            rec = len(p_tok & g_tok) / len(g_tok)
            f1s.append(2 * prec * rec / (prec + rec))
        accs.append(1.0 if llm_judge(p, g) else 0.0)
    return {'F1': sum(f1s)/len(f1s)*100, 'LLM Acc': sum(accs)/len(accs)*100}

Common pitfalls

• Models often generate verbose explanations despite prompts requesting direct answers, which artificially deflates F1 scores even when factual knowledge is correct.
• Cross-lingual and cross-modal performance drops are heavily influenced by ASR error rates (WER/CER), which vary significantly across languages (e.g., higher for French, Russian, Cantonese), confounding model capability with speech recognition quality.
• The LLM judge's exact prompt and configuration are deferred to the Appendix, making exact reproduction of LLM-based accuracy difficult without those details.

Evidence (verbatim from paper)

We evaluate using the F1 score and an LLM judge. Details on the judge selection and experimental setup are in the Appendix.

Citation

@misc{du2025ccfqa,
  title={CCFQA: A Benchmark for Cross-Lingual and Cross-Modal Speech and Text Factuality Evaluation},
  author={Du et al. (2025)},
  year={2025},
  note={arXiv:2508.07295}
}

• arXiv: 2508.07295