Qa4ie Eval

qa4ie-eval

QA4IE: A Question Answering based Framework for Information Extraction — Lin Qiu et al. (2018) (arXiv:1804.03396, 2018)

What this evaluates

Evaluates document-level information extraction by framing it as a question answering task. It probes a model's ability to extract cross-sentence relation triples from large documents using entity-relation queries and knowledge base alignment.

Datasets

• QA4IE — total ?; splits: train (-1), dev (-1), test (-1); repo https://github.com/SJTU-lqiu/QA4IE

Metrics

• Exact Match (EM) (primary) — range: [0, 1]
  • EM measures the percentage that the model prediction matches one of the ground truth answers exactly.
• F1-score (primary) — range: [0, 1]
  • F1-score measures the token-level overlap between the prediction and ground truth answers, calculated as 2 * (precision * recall) / (precision + recall).

Input / output format

Input: A document (or sentence) and a candidate entity-relation query (entity-property pair).

Output: A predicted answer span or sequence; outputs an 'eos' symbol or no answer if the confidence score is below a threshold δ.

Scoring recipe

def compute_metrics(pred, golds):
    em = 1.0 if pred.strip() in [g.strip() for g in golds] else 0.0
    pred_tokens = pred.split()
    gold_tokens = [g.split() for g in golds]
    intersection = sum(Counter(pred_tokens) & Counter(g) for g in gold_tokens)
    precision = intersection / len(pred_tokens) if pred_tokens else 0
    recall = intersection / len(gold_tokens[0]) if gold_tokens else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return em, f1

Common pitfalls

• The IE evaluation relies on confidence thresholds (Score_mul, Score_avg) to filter answers, causing precision/recall to vary drastically with the threshold.
• Baseline Open IE systems are evaluated on isolated sentences rather than full documents, creating an unfair comparison where baselines receive easier inputs.
• The benchmark assumes the first entity (subject) is known from the article title, so it does not evaluate entity recognition or linking.

Evidence (verbatim from paper)

Two metrics are introduced in the SQuAD dataset: Exact Match (EM) and F1-score. EM measures the percentage that the model prediction matches one of the ground truth answers exactly while F1-score measures the overlap between the prediction and ground truth answers. Our QA4IE benchmark also adopts these two metrics.

Citation

@misc{qiu2018qa4ie,
  title={QA4IE: A Question Answering based Framework for Information Extraction},
  author={Lin Qiu et al. (2018)},
  year={2018},
  note={arXiv:1804.03396}
}

• arXiv: 1804.03396