qhjqhj00 knesset-corpus-extraction-eval
The Knesset Corpus: An Annotated Corpus of Hebrew Parliamentary Proceedings — Sommer et al. (2024) (arXiv:2405.18115, 2024)
What this evaluates
Evaluates the accuracy and robustness of an automated extraction pipeline that converts raw Hebrew parliamentary documents into structured metadata, speaker lists, and text. It probes the system's ability to correctly parse dates, identify speakers, extract sentences, and match speaker names to an official database of Knesset members.
Datasets
- Knesset Corpus — total 44027; splits: evaluation (20)
Metrics
success_rate(primary) — range: percent- Proportion of extracted speaker names that correctly match the official MP database after manual verification, calculated as correct_matches / total_records_analyzed.
Input / output format
Input: Raw parliamentary document files containing Hebrew text, dates, and speaker mentions.
Output: Structured records containing document metadata (date), extracted text, identified speaker names, and matched MP database entries.
Scoring recipe
def score_success_rate(extracted_names, mp_db):
correct = 0
total = len(extracted_names)
for name in extracted_names:
clean = normalize(name)
if clean in mp_db and mp_db[clean] == assigned_mp:
correct += 1
return (correct / total) * 100
Common pitfalls
- The evaluation subset is very small (20 documents) compared to the full corpus (42k+ files), so results may not generalize to the remaining 85%.
- Name matching errors are heavily driven by orthographic variations between protocol text and the official database, causing false negatives that are hard to automate.
- Cleaning artifacts like '<' prefixes on speaker names are counted as errors but remain easily identifiable by humans, inflating error rates if not handled contextually.
Evidence (verbatim from paper)
Out of the total set, 285,790 records were automatically detected as correct since a match was found and the assigned name was identical to the clean name. We manually analyzed the remaining 400K records. About 38K records (5.52%) were found to be incorrect and were manually fixed. This reflects success rate of 94.48% .
Citation
@misc{sommer2024knesset,
title={The Knesset Corpus: An Annotated Corpus of Hebrew Parliamentary Proceedings},
author={Sommer et al. (2024)},
year={2024},
note={arXiv:2405.18115}
}
- arXiv: 2405.18115