Lm Eval Harness Benchmarks Eval

lm-eval-harness-benchmarks-eval

SlimPajama-DC: Understanding Data Combinations for LLM Training — Zhiqiang Shen et al. (2023) (arXiv:2309.10818, 2023)

What this evaluates

Evaluates generative language models on a suite of multiple-choice and open-ended benchmarks covering reasoning, commonsense, multitask proficiency, and truthfulness. It measures accuracy across diverse domains to assess generalization and the impact of data combination strategies.

Datasets

• AI2 Reasoning Challenge (ARC) — total ?; splits: test (-1)
• HellaSwag — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)
• TruthfulQA — total ?; splits: test (-1)
• BigBench — total ?; splits: test (-1)
• HumanEval — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted answers across all benchmark items. Computed as the number of matches between model predictions and gold labels divided by the total number of items.
• RRGS — range: [0, 1]
  • Risk of Random Guessing Score. Measures the variance of MMLU sub-item scores around the 25% random baseline using average L1 distance. Formula: RRGS = 1 - (1/N) * sum(|s_i - 0.25|) for i=1 to N.

Input / output format

Input: Multiple-choice questions or open-ended prompts with few-shot examples (0, 5, 10, or 25 shots depending on benchmark) provided via the Eleuther AI LM Evaluation Harness.

Output: Model-generated text completion or selected option.

Scoring recipe

def compute_accuracy(predictions, gold):
    return sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)

def compute_rrgs(sub_scores, baseline=0.25):
    n = len(sub_scores)
    avg_l1 = sum(abs(s - baseline) for s in sub_scores) / n
    return 1 - avg_l1

Common pitfalls

• TruthfulQA is initialized with 0-shot in the harness but consistently runs as a 6-shot task.
• RRGS evaluates MMLU sub-item variance around 25% to detect random guessing, not standard task accuracy.
• Lower training loss does not directly correlate with higher benchmark accuracy.

Evidence (verbatim from paper)

Since 25% in MMLU represents the baseline score for a guess, this metric evaluates the variance using average l1 distance around this base value across all sub-items. A larger variance would suggest a reduced likelihood of predictions resulting from mere chance. Given a MMLU score vector X of length N with sub-item scores s1, s2, ..., sn, RRGS can be formulated as: RRGS = 1 - (1/N) * sum(|si - 0.25|)

Citation

@misc{shen2023slimpajamadc,
  title={SlimPajama-DC: Understanding Data Combinations for LLM Training},
  author={Zhiqiang Shen et al. (2023)},
  year={2023},
  note={arXiv:2309.10818}
}

• arXiv: 2309.10818