Decipherment Eval

decipherment-eval

Deciphering Undersegmented Ancient Scripts Using Phonetic Prior — Luo et al. (2020) (arXiv:2010.11054, 2020)

What this evaluates

This benchmark evaluates a model's ability to decipher undersegmented ancient scripts by aligning unknown character sequences with known language stems. It probes phonological reasoning and unsupervised segmentation capabilities without relying on known language proximity or complete word boundaries.

Datasets

• Gothic — total 40518; splits: test (-1)
• Ugaritic — total 7353; splits: test (-1)
• Iberian — total 3466; splits: test (-1)

Metrics

• P@10 (primary) — range: [0, 1]
  • Precision at K measures the fraction of correct predictions among the top K candidates. A prediction is correct only if the predicted stem exactly matches the gold stem and the predicted character span is a strict prefix of the gold span.

Input / output format

Input: Unsegmented or undersegmented sequences of ancient script characters, optionally augmented with partial phonetic mappings for known characters.

Output: Ranked list of predicted stem-span pairs mapping the ancient script to known language stems.

Scoring recipe

def compute_p_at_k(predictions, gold, k=10):
    correct = 0
    for pred in predictions[:k]:
        if pred.stem == gold.stem and gold.span.startswith(pred.span):
            correct += 1
    return correct / k

Common pitfalls

• Matching is performed at the stem level rather than full words to account for historical linguistic drift.
• The predicted span must be a strict prefix of the gold span, not merely overlapping or equal.
• Undersegmentation is simulated by randomly inserting ground-truth whitespaces, so evaluation conditions vary by the preset whitespace ratio.

Evidence (verbatim from paper)

Metric We evaluate the model performance using precision at K (P@K) scores. The prediction (i.e., the stem-span pair) is considered correct if and only if the stem is correct and the span is the prefix of the ground truth. For instance, the ground truth for the Gothic word garda has the stem gard spanning the first four letters, matching the Old Norse stem garð. We only consider the prediction as correct if it correctly matches garð and the predicted span starts with the first letter.

Citation

@misc{luo2020deciphering,
  title={Deciphering Undersegmented Ancient Scripts Using Phonetic Prior},
  author={Luo et al. (2020)},
  year={2020},
  note={arXiv:2010.11054}
}

• arXiv: 2010.11054