Tat LLM Eval

tat-llm-eval

TAT-LLM: A Specialized Language Model for Discrete Reasoning over Tabular and Textual Data — Fengbin Zhu et al. (2024) (arXiv:2401.13223, 2024)

What this evaluates

Evaluates discrete reasoning capabilities over hybrid tabular and textual data, specifically focusing on financial question answering tasks that require arithmetic operations, counting, and span extraction.

Datasets

• FinQA — total ?; splits: train (-1), val (-1), test (-1)
• TAT-QA — total ?; splits: train (-1), val (-1), test (-1)
• TAT-DQA — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• EM (primary) — range: [0, 1]
  • Exact match between the predicted answer and the ground truth answer. For FinQA, this corresponds to Execution Accuracy.
• F1 — range: [0, 1]
  • Numeracy-focused macro-averaged F1 score measuring bag-of-words overlap between predicted and gold answers. It is forced to 0 if the predicted number does not exactly equal the ground truth number.

Input / output format

Input: A question accompanied by tabular data and textual paragraphs extracted from financial reports.

Output: A predicted numerical answer (and scale for TAT-LLM); zero-shot LLM evaluations omit scale prediction.

Scoring recipe

def score(predictions, golds):
    em = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    f1s = []
    for p, g in zip(predictions, golds):
        p_num = extract_number(p)
        g_num = extract_number(g)
        if p_num != g_num:
            f1s.append(0.0)
        else:
            p_tok, g_tok = set(p.lower().split()), set(g.lower().split())
            inter = len(p_tok & g_tok)
            union = len(p_tok | g_tok)
            f1s.append(2 * inter / union if union > 0 else 0.0)
    return em, sum(f1s) / len(f1s)

Common pitfalls

• Scale prediction is omitted for zero-shot LLM evaluations but included for TAT-LLM, leading to inconsistent comparison if not handled uniformly.
• The F1 score is numeracy-focused: it drops to 0 if the predicted number does not exactly match the ground truth, regardless of textual overlap.
• FinQA uses EM (Execution Accuracy) as its primary metric, while TAT-QA and TAT-DQA report both EM and F1, requiring dataset-specific metric tracking.

Evidence (verbatim from paper)

For TAT-QA and TAT-DQA datasets, we adopt the Exact Match (EM) and the numeracy-focused (macro-averaged) F1 score Zhu et al. ([2021], [2022]). Both two metrics measure the overlap between a bag-of-words representation of the gold and predicted answers. The numeracy-focused F1 score is set to 0 unless the predicted number is exactly equal to the ground truth. We use EM for FinQA, which is the same as the metric of Execution Accuracy originally used in FinQA Chen et al. ([2021]).

Citation

@misc{zhu2024tatllm,
  title={TAT-LLM: A Specialized Language Model for Discrete Reasoning over Tabular and Textual Data},
  author={Fengbin Zhu et al. (2024)},
  year={2024},
  note={arXiv:2401.13223}
}

• arXiv: 2401.13223