Caqa Attribution Eval

caqa-attribution-eval

Can LLMs Evaluate Complex Attribution in QA? Automatic Benchmarking using Knowledge Graphs — Nan Hu et al. (2024) (arXiv:2401.14640, 2024)

What this evaluates

Evaluates the quality and validity of citations in generated answers for complex question answering. It probes whether models can correctly classify attributions as supportive, insufficient, contradictory, or irrelevant, and assesses their ability to handle varying reasoning complexities.

Datasets

• CAQA — total 161174; splits: train (137211), test (23963)
• ACLE-Manual — total 850; splits: test (850)

Metrics

• micro-F1 (primary) — range: [0, 1]
  • Aggregates true positives, false positives, and false negatives across all classes to compute global precision and recall, then calculates F1 = 2 * (precision * recall) / (precision + recall). Explicitly used to handle the unbalanced distribution of attribution categories.

Input / output format

Input: Question, model-generated answer, and the specific citation/attribution text to be evaluated.

Output: One of four attribution labels: supportive, insufficient, contradictory, or irrelevant.

Scoring recipe

def compute_micro_f1(preds, gold):
    tp = fp = fn = 0
    for p, g in zip(preds, gold):
        if p == g:
            tp += 1
        elif p != g:
            fp += 1
            fn += 1
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0

Common pitfalls

• The dataset has a highly unbalanced distribution of attribution categories, making macro-F1 misleading; micro-F1 is explicitly required.
• Zero-shot evaluators often fail to detect subtle attribution errors (e.g., insufficient vs. supportive) without fine-tuning on the CAQA benchmark.
• Human annotation consistency on the 600 test samples may differ from automatic categories, requiring careful alignment when comparing automated vs. human judgments.

Evidence (verbatim from paper)

For all experiment results in this work, we report the F1 score for the performance on each attribution category (or each complexity level) calculated according its precision and recall, and the micro-F1 score for the overall performance. The micro-F1 score is used because the distribution of attribution categories (or complexity levels) is unbalanced.

Citation

@misc{hu2024caqa,
  title={Can LLMs Evaluate Complex Attribution in QA? Automatic Benchmarking using Knowledge Graphs},
  author={Nan Hu et al. (2024)},
  year={2024},
  note={arXiv:2401.14640}
}

• arXiv: 2401.14640