# Caqa Attribution Eval

> Evaluates the quality and validity of citations in generated answers for complex question answering. It probes whether models can correctly classify attributions as supportive, insufficient, contradictory, or irrelevant, and assesses their ability to handle varying reasoning complexities. Use when the user wants to benchmark on CAQA, ACLE-Manual, or asks about evaluating this task. Reports micro-F1.

- Skill: `qhjqhj00/caqa-attribution-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/caqa-attribution-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/caqa-attribution-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/caqa-attribution-eval

---


# caqa-attribution-eval

> Can LLMs Evaluate Complex Attribution in QA? Automatic Benchmarking using Knowledge Graphs — Nan Hu et al. (2024) (arXiv:2401.14640, 2024)

## What this evaluates

Evaluates the quality and validity of citations in generated answers for complex question answering. It probes whether models can correctly classify attributions as supportive, insufficient, contradictory, or irrelevant, and assesses their ability to handle varying reasoning complexities.

## Datasets

- **CAQA** — total 161174; splits: train (137211), test (23963)
- **ACLE-Manual** — total 850; splits: test (850)

## Metrics

- `micro-F1` **(primary)** — range: [0, 1]
  - Aggregates true positives, false positives, and false negatives across all classes to compute global precision and recall, then calculates F1 = 2 * (precision * recall) / (precision + recall). Explicitly used to handle the unbalanced distribution of attribution categories.

## Input / output format

**Input**: Question, model-generated answer, and the specific citation/attribution text to be evaluated.

**Output**: One of four attribution labels: supportive, insufficient, contradictory, or irrelevant.

## Scoring recipe

```python
def compute_micro_f1(preds, gold):
    tp = fp = fn = 0
    for p, g in zip(preds, gold):
        if p == g:
            tp += 1
        elif p != g:
            fp += 1
            fn += 1
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0
```

## Common pitfalls

- The dataset has a highly unbalanced distribution of attribution categories, making macro-F1 misleading; micro-F1 is explicitly required.
- Zero-shot evaluators often fail to detect subtle attribution errors (e.g., insufficient vs. supportive) without fine-tuning on the CAQA benchmark.
- Human annotation consistency on the 600 test samples may differ from automatic categories, requiring careful alignment when comparing automated vs. human judgments.

## Evidence (verbatim from paper)

> For all experiment results in this work, we report the F1 score for the performance on each attribution category (or each complexity level) calculated according its precision and recall, and the micro-F1 score for the overall performance. The micro-F1 score is used because the distribution of attribution categories (or complexity levels) is unbalanced.

## Citation

```bibtex
@misc{hu2024caqa,
  title={Can LLMs Evaluate Complex Attribution in QA? Automatic Benchmarking using Knowledge Graphs},
  author={Nan Hu et al. (2024)},
  year={2024},
  note={arXiv:2401.14640}
}
```

- arXiv: 2401.14640