Unknown Token Rate

unknown_token_rate

Since the Scientific Literature Is Multilingual, Our Models Should Be Too — Ebrahimi et al. (2024) (arXiv:2403.18251, 2024)

What this evaluates

Evaluates scientific document representation models on multilingual abstracts by measuring tokenization coverage, language modeling perplexity, and embedding quality relative to citation networks. It probes whether models can meaningfully process non-Latin scripts and low-resource languages without degrading to English-only or graph-based heuristics.

Datasets

• Scientific abstracts sample — total ?; splits: (unstated)

Metrics

• unknown_token_rate (primary) — range: percent
  • Percentage of input tokens that are tokenized as UNK per language.
• pseudo_perplexity (PPPL) — range: other
  • Model score as defined by Salazar et al. (2020), measuring intrinsic language modeling performance on the given text.
• centroid_cosine_similarity — range: [0, 1]
  • Cosine similarity between a document's embedding vector and the centroid vector approximated by summing its reference vectors.

Input / output format

Input: Abstract text (tokenized into subword/character tokens depending on the model).

Output: Model embeddings, perplexity scores, or UNK token counts per language.

Scoring recipe

def compute_metrics(tokenized_abstract, model, references):
    # 1. Unknown Token Rate
    unk_count = sum(1 for t in tokenized_abstract if t == 'UNK')
    unk_rate = (unk_count / len(tokenized_abstract)) * 100
    
    # 2. Pseudo-Perplexity (PPPL)
    pppl = model_score_per_Salazar2020(tokenized_abstract)
    
    # 3. Centroid Cosine Similarity
    doc_vec = model.encode(document)
    ref_vecs = [model.encode(ref) for ref in references]
    centroid_vec = sum(ref_vecs)
    cosine_sim = cosine_similarity(doc_vec, centroid_vec)
    
    return unk_rate, pppl, cosine_sim

Common pitfalls

• Assuming low perplexity indicates strong multilingual capability without first checking the unknown token rate, as high UNK rates can artificially lower perplexity by masking non-Latin scripts.
• Ignoring sequence length limits: English translations appended to non-Latin abstracts may be truncated because UNK tokens from the source script consume the context window.
• Directly comparing text-based models to graph-based methods (e.g., ProNE) without accounting for architectural differences in how they leverage citation data.

Evidence (verbatim from paper)

To quantify the impact of using only a Latin-script vocabulary, we calculate the average percentage of UNK tokens contained in the tokenized abstracts for each language in our sample... To intrinsically evaluate performance on languages without a large number of UNK tokens, we calculate a pseudo-perplexity (PPPL), or model score Salazar et al. ([2020])... This assumption approximates a document vector as the sum of its references. The boxplots in Figure[3] use cosines to compare documents vectors with the centroids of their references.

Citation

@misc{ebrahimi2024multilingualscientific,
  title={Since the Scientific Literature Is Multilingual, Our Models Should Be Too},
  author={Ebrahimi et al. (2024)},
  year={2024},
  note={arXiv:2403.18251}
}

• arXiv: 2403.18251