Chatqa Conversational QA Eval

chatqa-conversational-qa-eval

ChatQA: Surpassing GPT-4 on Conversational QA and RAG — Liu et al. (2024) (arXiv:2401.10225, 2024)

What this evaluates

Evaluates multi-turn conversational question answering over both long and short documents, including text-only and tabular contexts. It probes a model's ability to retrieve relevant context, handle topic switching, perform arithmetic reasoning, and correctly identify unanswerable queries.

Datasets

• Doc2Dial — total ?; splits: test (-1)
• QuAC — total ?; splits: test (-1)
• QReCC — total ?; splits: test (-1)
• TopiOCQA — total ?; splits: test (-1)
• INSCIT — total ?; splits: test (-1)
• CoQA — total ?; splits: test (-1)
• DoQA — total ?; splits: test (-1)
• ConvFinQA — total ?; splits: test (-1)
• SQA — total ?; splits: test (-1)
• HybridDial — total ?; splits: test (-1)

Metrics

• Average F1/EM across 10 datasets (primary) — range: percent
  • The arithmetic mean of per-dataset scores. F1 score is used for 9 datasets, while Exact Match is used for ConvFinQA. For ConvFinQA, if the model outputs an arithmetic formula, its result is computed via a calculator before comparing to the gold answer. Scores are reported on a 0-100 scale.

Input / output format

Input: Multi-turn conversational dialogue history with a user query. For long-document datasets, the input is augmented with top-5 or top-20 retrieved document chunks. For short-document datasets, the full document (text or table) is provided directly.

Output: A natural language answer to the user's query, or a statement indicating the query is unanswerable.

Scoring recipe

def evaluate(dataset_name, predictions, golds):
    if dataset_name == 'ConvFinQA':
        # Exact match or evaluate generated arithmetic formula
        return exact_match(predictions, golds)
    else:
        return f1_score(predictions, golds)

# Compute per-dataset score
scores = [evaluate(ds, preds[ds], golds[ds]) for ds in DATASETS]
# Headline metric is the average across all 10 datasets
final_metric = sum(scores) / len(scores)

Common pitfalls

• ConvFinQA requires Exact Match instead of F1, and mandates evaluating any generated arithmetic formulas via a calculator before comparison.
• Long-document datasets require consistent chunking (~300 words) and top-k retrieval; failing to use the same retrieved chunks for all baselines breaks fair comparison.
• TopiOCQA and INSCIT use original document segmentation rather than the ~300-word chunking applied to Doc2Dial, QuAC, and QReCC.

Evidence (verbatim from paper)

Given that F1 score is the most commonly used automatic metric to assess QA models, we use it for all datasets except for ConvFinQA. In ConvFinQA, we follow Chen et al. (2022a) to use exact match metric since the answers in ConvFinQA are about extracting numbers from documents as well as arithmetic calculations. Hence, the answer only makes sense when it is exactly the same as the answer. When models generate the arithmetic formula, we will calculate its final result based on a calculator and compare it with the gold answer.

Citation

@misc{liu2024chatqa,
  title={ChatQA: Surpassing GPT-4 on Conversational QA and RAG},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2401.10225}
}

• arXiv: 2401.10225