Script Identification Eval

script-identification-eval

GlotScript: A Resource and Tool for Low Resource Writing System Identification — Kargaran et al. (2023) (arXiv:2309.13320, 2023)

What this evaluates

Evaluates the accuracy of a script identification tool on multilingual web corpora by checking if the predicted writing system matches the admissible scripts for the corpus's assigned language. It also measures script representation coverage in multilingual LLM tokenizers.

Datasets

• Multilingual C4 (mC4) — total ?; splits: test (-1)
• OSCAR 22.01 — total ?; splits: test (-1)

Metrics

• ACC (primary) — range: [0, 1]
  • Proportion of sentences where the predicted main script is in the set of admissible scripts for the language provided by corpus metadata.

Input / output format

Input: Text sentences from multilingual web corpora (mC4, OSCAR) paired with corpus-provided language metadata (ISO 639 code, optionally ISO 15924 script).

Output: A single ISO 15924 script code representing the dominant writing system in the input sentence.

Scoring recipe

def compute_acc(sentences, langs, admissible_map):
    correct = 0
    for s, lang in zip(sentences, langs):
        pred = get_main_script(s)
        allowed = admissible_map.get(lang, [])
        if pred in allowed:
            correct += 1
    return correct / len(sentences)

Common pitfalls

• Overlooking the metadata script override rule: if corpus metadata specifies a script (e.g., bg-Latin), only that script is admissible, not the full GlotScript-R list.
• Not applying the 1000-sentence per language sampling filter, which skews results toward languages with abundant data.
• Confusing ACC with ACC70/ACC50, which restrict evaluation to sentences exceeding 70% or 50% of the maximum length in that language subset.

Evidence (verbatim from paper)

We determine for each sentence of the corpus whether it is a match or a mismatch and then report the proportion of errors. ... ACC: accuracy, i.e., the proportion of sentences for which the script identified by GlotScript-T is one of the admissible scripts (according to GlotScript-R) of the language provided by corpus metadata for the sentence.

Citation

@misc{kargaran2023glotscript,
  title={GlotScript: A Resource and Tool for Low Resource Writing System Identification},
  author={Kargaran et al. (2023)},
  year={2023},
  note={arXiv:2309.13320}
}

• arXiv: 2309.13320