QA Zre Eval

qa-zre-eval

FabricQA-Extractor: A Question Answering System to Extract Information from Documents using Natural Language Questions — Wang et al. (2024) (arXiv:2408.09226, 2024)

What this evaluates

Evaluates the ability of a question-answering system to extract missing objects from partially filled relational tuples by searching through a large document corpus. It probes schema-aware information extraction and the model's capacity to leverage relational coherence across multiple questions.

Datasets

• QA-ZRE — total 1192; splits: train (-1), dev (-1), test (-1)

Metrics

• Exact Match (EM) (primary) — range: [0, 1]
  • Takes value 1 when the normalized predicted answer exactly matches the normalized ground truth answer, and 0 otherwise. Normalization removes articles, punctuation, and converts characters to lowercase using the same procedure for both.
• F1 Score (F1) — range: [0, 1]
  • Token-level F1 score computed as the harmonic mean of precision (#correct tokens in answer / tokens in predicted answer) and recall (#correct tokens in answer / tokens in ground truth answer).

Input / output format

Input: A natural language question, a partially filled relational tuple (subject and relationship), and a candidate passage (or a set of retrieved passages from a large corpus) containing the target information.

Output: A single predicted answer string corresponding to the missing object in the relational tuple.

Scoring recipe

def compute_metrics(predictions, golds):
    em_scores = []
    f1_scores = []
    for pred, gold in zip(predictions, golds):
        pred_norm = normalize(pred)
        gold_norm = normalize(gold)
        em_scores.append(1.0 if pred_norm == gold_norm else 0.0)
        pred_tokens = pred_norm.split()
        gold_tokens = gold_norm.split()
        common = Counter(pred_tokens) & Counter(gold_tokens)
        num_same = sum(common.values())
        if num_same == 0:
            f1_scores.append(0.0)
            continue
        precision = num_same / len(pred_tokens)
        recall = num_same / len(gold_tokens)
        f1_scores.append(2 * precision * recall / (precision + recall))
    return {'EM': sum(em_scores)/len(em_scores), 'F1': sum(f1_scores)/len(f1_scores)}

Common pitfalls

• Normalization must be applied identically to both predictions and ground truth before comparison, or EM/F1 scores will be artificially low.
• The benchmark evaluates retrieval-augmented QA over a massive corpus (35M+ passages), not just closed-book reading comprehension; passing the wrong passage will yield 0 score regardless of the reader's quality.
• Aggregated metrics are computed per table/relationship, not globally across all questions, which can skew overall performance if some tables have many questions.

Evidence (verbatim from paper)

Exact Match (EM): For each question and answer pair, this metric takes as value 1 when the answer exactly matches the ground truth answer, and 0 when it does not. Both ground truth and the provided answer are first normalized (removing articles, punctuation, convert characters to lower case) using the same procedure.

F1 Score (F1): Because the EM metric is sensitive to the specific tokens in the answer (e.g., "the 4th of July" vs "July, 4th"), the F1 score is computed at the token level, using the precision (#correct tokens in answer / tokens in predicted answer) and recall (#correct tokens in answer / tokens in ground truth answer).

Citation

@misc{wang2024fabricqaextractor,
  title={FabricQA-Extractor: A Question Answering System to Extract Information from Documents using Natural Language Questions},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2408.09226}
}

• arXiv: 2408.09226