Bib Ref Parser Eval

bib-ref-parser-eval

Machine Learning vs. Rules and Out-of-the-Box vs. Retrained: An Evaluation of Open-Source Bibliographic Reference and Citation Parsers — Tkaczyk et al. (2018) (arXiv:1802.01168, 2018)

What this evaluates

Evaluates open-source bibliographic reference and citation parsers on their ability to extract structured metadata fields (author, source, year, volume, issue, page, organization) from raw citation text. Compares machine learning-based versus rule-based approaches, and assesses the impact of domain-specific retraining.

Datasets

• Unspecified bibliographic dataset — total ?; splits: test (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: F1 = 2 * (precision * recall) / (precision + recall). Calculated per metadata field (author, source, year, volume, issue, page, organization) and averaged for overall scores.

Input / output format

Input: Raw bibliographic reference or citation text strings.

Output: Structured metadata fields: author, source, year, volume, issue, page, and organization.

Scoring recipe

def compute_f1(preds, golds):
    tp = sum(1 for p, g in zip(preds, golds) if p == g)
    fp = sum(1 for p, g in zip(preds, golds) if p != g and p is not None)
    fn = sum(1 for p, g in zip(preds, golds) if p is None and g is not None)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

Common pitfalls

• Organization field is consistently unextracted by all evaluated systems, so it is often excluded from overall scoring.
• Rule-based parsers show high precision but extremely low recall, skewing average performance if only precision is considered.
• F1 is computed per metadata field type before averaging, not just on the entire citation string.

Evidence (verbatim from paper)

Each cell in the table gives precision, recall and F1 values, respectively. For each combination (metadata type, metric) the best result is bolded. We do not give the results for organization, as none of the systems is able to extract this field. Measured with F1, the best performing out-of-the-box tools are: GROBID (F1 0.89), followed by CERMINE (F1 0.83) and ParsCit (F1 0.75).

Citation

@misc{tkaczyk2018machine,
  title={Machine Learning vs. Rules and Out-of-the-Box vs. Retrained: An Evaluation of Open-Source Bibliographic Reference and Citation Parsers},
  author={Tkaczyk et al. (2018)},
  year={2018},
  note={arXiv:1802.01168}
}

• arXiv: 1802.01168