Language Ranker

language-ranker

Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages — Li et al. (2024) (arXiv:2404.11553, 2024)

What this evaluates

This metric probes an LLM's internal representation quality and cross-lingual alignment by measuring how closely the embedding space of a target language clusters around an English baseline. It quantifies multilingual capability and pre-training data imbalance by computing similarity scores across specific transformer layers.

Datasets

• ARC — total 7787; splits: test (7787); repo https://github.com/nlp-uoregon/mlmm-evaluation
• MMLU — total ?; splits: test (-1)

Metrics

• Language Ranker (primary) — range: [0, 1]
  • Cosine similarity between the mean embedding of a target language and the mean embedding of English, computed at specific transformer layers (5, 10, 15, 20, 25).

Input / output format

Input: Internal hidden state representations (embeddings) of an LLM for a set of sentences in a target language and a baseline set in English.

Output: A scalar cosine similarity score per language per model layer, ranging from 0 to 1.

Scoring recipe

def compute_language_ranker(model, target_texts, english_texts, layers=[5,10,15,20,25]):
    scores = {}
    for layer in layers:
        t_emb = model.get_embeddings(target_texts, layer)
        e_emb = model.get_embeddings(english_texts, layer)
        t_mean = t_emb.mean(dim=0)
        e_mean = e_emb.mean(dim=0)
        scores[layer] = cosine_sim(t_mean, e_mean)
    return scores

Common pitfalls

• The metric is intrinsic and measures representation alignment, not direct downstream task performance.
• Scores are heavily influenced by pre-training corpus proportions, so high similarity may reflect data abundance rather than actual reasoning capability.
• The choice of English baseline sentences and specific transformer layers significantly impacts the resulting scores.

Evidence (verbatim from paper)

From Figure[1], we can observe that high-resource languages have representations more similar to English, whereas low-resource languages show less similarity. Specifically, German, Spanish, French, and Malay generally maintain cosine similarity scores above 0.6, with Spanish and French often showing the highest scores, indicating that these languages are better represented in the models’ embeddings. In contrast, low-resource languages, such as Igbo, Kazakh, Kannada, Oriya, and Turkmen, display significantly lower cosine similarity scores, often below 0.4. These results show the disparities in performance across languages and highlights the utility of the Language Ranker in quantifying these differences robustly.

Citation

@misc{li2024langueranker,
  title={Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2404.11553}
}

• arXiv: 2404.11553