Pariksha Eval

pariksha-eval

PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data — Watts et al. (2024) (arXiv:2406.15053, 2024)

What this evaluates

Evaluates multilingual and multi-cultural LLM performance across 10 Indic languages using culturally nuanced prompts. It measures model quality via pairwise comparisons (Elo ratings) and direct assessment scores, while also analyzing human-LLM evaluator agreement and various biases (position, verbosity, self-bias).

Datasets

• PARIKSHA — total 90000; splits: test (-1)

Metrics

• Elo rating (primary) — range: other
  • Pairwise comparison metric based on the MLE Elo rating method. Models are ranked by win/loss/tie rates in head-to-head battles across languages.
• Direct Assessment score (primary) — range: [0, 1]
  • Average score across all query-response pairs for a model, evaluating metrics like Linguistic Acceptability (LA) and Task Quality (TQ).
• Fleiss Kappa ($\kappa$) — range: [-1, 1]
  • Measures inter-annotator agreement for categorical items, calculated at a per-datapoint level for both human-human and human-LLM evaluations.
• Kendall Tau ($\tau$) — range: [-1, 1]
  • Rank correlation coefficient measuring agreement between human and LLM-evaluator leaderboards.

Input / output format

Input: Pairwise: prompt + response A + response B. Direct Assessment: prompt + response A. Safety: prompt + response.

Output: Pairwise: 'A', 'B', or 'tie'. Direct Assessment: numerical score for Linguistic Acceptability (LA) and Task Quality (TQ), plus hallucination flag. Safety: binary problematic/not problematic.

Scoring recipe

def compute_elo(wins, losses, ties, total_games):
    win_rate = (wins + 0.5 * ties) / total_games
    # MLE Elo rating update based on win_rate
    return elo_rating

def compute_da_score(scores):
    return sum(scores) / len(scores)

def compute_fleiss_kappa(raters, categories):
    # Standard Fleiss Kappa formula over per-datapoint annotations
    return kappa

def compute_kendall_tau(rankings_h, rankings_l):
    return kendalltau(rankings_h, rankings_l)

Common pitfalls

• LLM evaluators pick fewer ties and are more decisive than humans, inflating win rates and skewing leaderboard rankings.
• Direct assessment yields significantly lower human-LLM agreement than pairwise, especially for culturally nuanced prompts and low-resource Indic languages like Bengali and Odia.
• Heuristic toxicity word-matching underperforms LLM evaluators due to limited stem forms and inability to catch contextual toxicity.

Evidence (verbatim from paper)

We compute the Percentage Agreement (PA) and Fleiss Kappa ($\kappa$) score which are calculated at a per-datapoint level as well as the general agreement between the leaderboards using Kendall’s Tau ($\tau$).

Citation

@misc{watts2024pariksha,
  title={PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data},
  author={Watts et al. (2024)},
  year={2024},
  note={arXiv:2406.15053}
}

• arXiv: 2406.15053