Clas Bench Eval

clas-bench-eval

CLaS-Bench: A Cross-Lingual Alignment and Steering Benchmark — Gurgurov et al. (2026) (arXiv:2601.08331, 2026)

What this evaluates

Evaluates the effectiveness of various language steering methods in large language models across 32 languages. It measures how well interventions force the model to output in a target language while preserving the semantic relevance of the response.

Datasets

• CLaS-Bench — total 70; splits: test (70); repo https://github.com/d-gurgurov/CLaS-Bench

Metrics

• language forcing — range: percent
  • Percentage of generated responses that correctly match the target language, typically evaluated via automated language identification or manual verification.
• semantic relevance — range: percent
  • Quality score of the generated response's content relative to the prompt, assessed by a judge model or human annotators.
• steering score (primary) — range: percent
  • Harmonic mean of language forcing and semantic relevance: 2 * (forcing * relevance) / (forcing + relevance).

Input / output format

Input: A question prompt, optionally with language specification instructions, processed through a specified steering method (prompting or representation intervention).

Output: The model's generated text response.

Scoring recipe

def compute_steering_score(forcing_rate, relevance_score):
    if forcing_rate + relevance_score == 0:
        return 0.0
    return 2 * (forcing_rate * relevance_score) / (forcing_rate + relevance_score)

Common pitfalls

• Steering strength parameter α is not standardized across all methods in the provided setup, making direct comparison sensitive to hyperparameter tuning.
• The exact implementation of the 'Judge Quality' metric for semantic relevance is not detailed in this section, requiring external documentation for reproducibility.
• Language forcing evaluation relies on automated language ID or manual checks not specified here, which can introduce bias for low-resource or code-switched outputs.

Evidence (verbatim from paper)

It evaluates steering methods—including residual-stream DiffMean, probe-derived directions, and language-specific neurons—on two axes: language forcing (output switches to target language) and semantic relevance, combined into a harmonic-mean steering score.

Citation

@misc{gurgurov2026clasbench,
  title={CLaS-Bench: A Cross-Lingual Alignment and Steering Benchmark},
  author={Gurgurov et al. (2026)},
  year={2026},
  note={arXiv:2601.08331}
}

• arXiv: 2601.08331