Cross Linguistic Activation Eval

cross-linguistic-activation-eval

Uncovering Cross-Linguistic Disparities in LLMs using Sparse Autoencoders — Richmond Sin Jing Xuan, Jalil Huseynov, Yang Zhang (2025) (arXiv:2507.18918, 2025)

What this evaluates

Evaluates cross-linguistic disparities in LLMs by measuring activation gaps via Sparse Autoencoders and benchmark performance across high-resource and medium-to-low resource languages. It probes whether surface-level embedding alignment guarantees equitable model behavior and tests if activation-level fine-tuning can close performance gaps without degrading English capabilities.

Datasets

• ARC-Challenge — total ?; splits: test (-1); HF allenai/ai2_arc
• HellaSwag — total ?; splits: test (-1); HF openai/hellaswag
• MMLU — total ?; splits: test (-1); HF cais/mmlu

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of correctly answered questions on benchmark tasks.
• cosine similarity — range: [0, 1]
  • Cosine of the angle between two residual activation vectors, measuring embedding alignment.
• activation gap — range: percent
  • Percentage difference in mean SAE activation levels between a target language and English across layers.

Input / output format

Input: Benchmark prompts (10-shot for ARC-Challenge and HellaSwag) and multilingual text pairs for embedding similarity analysis.

Output: Model predictions for multiple-choice questions; SAE activation vectors across 26 layers for each language.

Scoring recipe

def compute_accuracy(preds, gold):
    return sum(p == g for p, g in zip(preds, gold)) / len(gold)

def compute_cosine_sim(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def compute_activation_gap(lang_a, lang_b):
    return abs(np.mean(lang_a) - np.mean(lang_b)) / np.mean(lang_a)

Common pitfalls

• High embedding similarity does not guarantee strong benchmark performance or equitable activation dynamics.
• Activation gaps vary significantly across layers, peaking in early layers but persisting in deeper layers.
• Fine-tuning must be evaluated for both target language gains and English performance retention to avoid catastrophic forgetting.

Evidence (verbatim from paper)

Fine-tuning resulted in a 1.44-point increase in ARC-C (Malayalam) accuracy, translating to a 5.47% improvement, demonstrating the positive impact of activation alignment on downstream tasks

Citation

@misc{sinjingxuan2025uncovering,
  title={Uncovering Cross-Linguistic Disparities in LLMs using Sparse Autoencoders},
  author={Richmond Sin Jing Xuan, Jalil Huseynov, Yang Zhang (2025)},
  year={2025},
  note={arXiv:2507.18918}
}

• arXiv: 2507.18918