qhjqhj00 turkish-sseg-eval
Mukayese: Turkish NLP Strikes Back — Safaya et al. (2022) (arXiv:2203.01215, 2022)
What this evaluates
Evaluates the detection of sentence boundaries in Turkish text across diverse domains (scientific abstracts, news, social media). It tests robustness to formatting variations and punctuation absence.
Datasets
- trseg-41 — total 10700; splits: train (-1), dev (-1), test (-1); repo https://github.com/alisafaya/mukayese
Metrics
F1-score(primary) — range: [0, 1]- Segment-level F1 score computed over predicted vs gold sentence boundaries.
Precision— range: [0, 1]- Segment-level precision for sentence boundary detection.
Recall— range: [0, 1]- Segment-level recall for sentence boundary detection.
Input / output format
Input: Raw text documents or social media posts.
Output: List of sentence boundary indices or segmented sentences.
Scoring recipe
def score_sseg(predictions, gold):
pred_segments = model.predict_boundaries(text)
gold_segments = gold_boundaries
# Compute segment-level metrics
p = len(set(pred_segments) & set(gold_segments)) / len(pred_segments) if pred_segments else 0
r = len(set(pred_segments) & set(gold_segments)) / len(gold_segments) if gold_segments else 0
f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0
return f1, p, r
Common pitfalls
- Evaluation is done on the entire test set without removing ambiguous boundaries, unlike some prior work.
- Corrupted test sets (lowercased/punctuation removed) are used to stress-test robustness, which drastically lowers scores.
- Single annotator labels may introduce subjectivity not captured by automated metrics.
Evidence (verbatim from paper)
Our evaluation procedure is based on the metrics F1 score, Precision, Recall for each segment. Unlike Wicks and Post (2021), we evaluate our models on the entire test set, without removing sentences with ambiguous boundaries.
Citation
@misc{safaya2022mukayese,
title={Mukayese: Turkish NLP Strikes Back},
author={Safaya et al. (2022)},
year={2022},
note={arXiv:2203.01215}
}
- arXiv: 2203.01215