Schema To JSON Eval

schema-to-json-eval

Schema-Driven Information Extraction from Heterogeneous Tables — Bai et al. (2023) (arXiv:2305.14336, 2023)

What this evaluates

Evaluates the ability of language models to extract structured information from heterogeneous tables (text, LaTeX, HTML, CSV, XML) using only a human-authored JSON schema as supervision. It probes schema-driven information extraction, testing attribute prediction accuracy across diverse domains and input formats without domain-specific labeled data.

Datasets

• MlTables — total ?; splits: train (-1), test (-1)
• ChemTables — total ?; splits: train (-1), test (-1)
• DisCoMat — total ?; splits: train (-1), test (-1)
• SWDE — total ?; splits: train (-1), test (-1)

Metrics

• Table-F1 (primary) — range: [0, 1]
  • Harmonic mean of attribute-level precision and recall. Precision is the ratio of correctly predicted attributes to total predicted attributes. Reported as macro-averaged across tables.
• Token-level F1 — range: [0, 1]
  • Token-level F1 score for attribute value matching. A prediction is correct if the score exceeds a dataset-specific threshold tuned on the dev set to maximize alignment with human judgments.
• Exact Match (EM) — range: [0, 1]
  • Exact match of predicted attribute values against gold values.
• Tuple-F1 — range: [0, 1]
  • Exact match of a predicted 4-element tuple against the gold tuple.
• Page-F1 — range: [0, 1]
  • Fraction of pages where all attributes are accurately predicted.

Input / output format

Input: Heterogeneous table data (text, LaTeX, HTML, CSV, XML) paired with a human-authored JSON schema template. Optional supplementary text (headers, captions) may be included.

Output: A JSON object conforming to the provided schema, containing extracted attribute values for each table.

Scoring recipe

def compute_table_f1(preds, golds):
    correct = 0
    total_pred = 0
    total_gold = 0
    for p, g in zip(preds, golds):
        matched = [k for k in p if k in g and token_f1(p[k], g[k]) > THRESHOLD]
        correct += len(matched)
        total_pred += len(p)
        total_gold += len(g)
    prec = correct / total_pred if total_pred else 0
    rec = correct / total_gold if total_gold else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) else 0

Common pitfalls

• Removing table captions surprisingly improves performance because low-quality or non-specific captions confuse the model.
• SWDE evaluation focuses on identifying the correct HTML node containing the attribute value rather than exact text span matching, requiring token-level F1 for node selection.
• The threshold for token-level F1 is dataset-specific and must be tuned on the development set to maximize alignment with human judgments.

Evidence (verbatim from paper)

We introduce Table-F1, a new metric gauging overall attribute prediction performance within a table, for our two proposed datasets. Table-F1 represents the harmonic mean of precision and recall, with precision being the ratio of correctly predicted attributes to total predicted attributes. At the attribute level, we consider two metrics: token-level F1 and exact match (EM). For DisCoMat and SWDE, we use the metrics specified in the original papers. In the case of DisCoMat, we report Tuple-F1, where a predicted 4-element tuple is considered correct only if it exactly matches with the gold tuple. For SWDE, we report Page-F1, which measures the number of pages where the attributes are accurately predicted.

Citation

@misc{bai2023schema,
  title={Schema-Driven Information Extraction from Heterogeneous Tables},
  author={Bai et al. (2023)},
  year={2023},
  note={arXiv:2305.14336}
}

• arXiv: 2305.14336