Compmix Eval

compmix-eval

CompMix: A Benchmark for Heterogeneous Question Answering — Christmann et al. (2023) (arXiv:2306.12235, 2023)

What this evaluates

Probes a model's ability to perform heterogeneous question answering by integrating information from multiple sources (knowledge bases, text, tables, infoboxes) across diverse domains and complex question intents. It specifically tests whether systems can fuse complementary structured and unstructured data to answer self-contained, human-generated questions.

Datasets

• CompMix — total 9410; splits: train (4966), dev (1680), test (2764)

Metrics

• answer exact match (primary) — range: [0, 1]
  • Compare the model's predicted answer to the gold answer. If the gold answer is a Wikidata entity identifier, check for ID equality. If it is a plaintext string or normalized date, check for exact string equality. The final score is the fraction of correctly matched answers.

Input / output format

Input: A self-contained, human-generated question spanning one of five domains (books, movies, music, TV series, soccer), optionally with entity markup. The model must retrieve and fuse information from Wikidata, Wikipedia text, tables, and infoboxes.

Output: An answer formatted as a Wikidata entity identifier, a plaintext string, or a normalized date.

Scoring recipe

def compute_answer_exact_match(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        pred = pred.strip()
        gold = gold.strip()
        if gold.startswith('Q') and len(gold) >= 2:  # Wikidata ID
            correct += (pred == gold)
        else:  # plaintext or normalized date
            correct += (pred == gold)
    return correct / len(golds) if golds else 0.0

Common pitfalls

• Questions require integrating multiple heterogeneous sources (KB, text, tables, infoboxes) rather than relying on a single information source.
• A significant fraction of questions involve long-tail entities, which often causes LLMs to fail due to missing pre-training knowledge.
• Answer formats vary (Wikidata IDs, plaintext, dates), requiring flexible matching logic rather than a single string comparison.

Evidence (verbatim from paper)

Answers are Wikidata entity identifiers (text labels are also provided), plaintext strings, or normalized dates. This enables consistent evaluation across extractive and generative answering models.

Citation

@misc{christmann2023compmix,
  title={CompMix: A Benchmark for Heterogeneous Question Answering},
  author={Christmann et al. (2023)},
  year={2023},
  note={arXiv:2306.12235}
}

• arXiv: 2306.12235