Skywork Benchmark Eval

skywork-benchmark-eval

Skywork: A More Open Bilingual Foundation Model — Wei et al. (2023) (arXiv:2310.19341, 2023)

What this evaluates

Evaluates bilingual foundation models on general knowledge, Chinese domain-specific reasoning, mathematical problem-solving, and language modeling capabilities using standardized benchmarks and custom held-out text corpora.

Datasets

• MMLU — total ?; splits: test (-1)
• CEVAL — total 13948; splits: test (13948)
• CMMLU — total ?; splits: test (-1)
• GSM8K — total 8500; splits: test (8500)
• Custom Chinese LM Testset — total ?; splits: test (-1)

Metrics

• 5-shot accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions across MMLU, CEVAL, and CMMLU. Computed as (number of correct predictions / total instances) * 100. Evaluated with 5 few-shot examples per prompt.
• 8-shot accuracy — range: percent
  • Percentage of correctly solved grade-school math word problems in GSM8K. Computed as (number of correct predictions / total instances) * 100. Evaluated with 8 few-shot examples per prompt.
• perplexity — range: other
  • Exponential of the average negative log-likelihood of the ground truth tokens over the custom Chinese language modeling testset. Lower values indicate better language modeling capability.

Input / output format

Input: Multiple-choice questions or math word problems accompanied by 5 or 8 few-shot examples for accuracy benchmarks; unlabeled natural text documents published after September 1, 2023 for language modeling.

Output: Model's predicted answer option or generated text sequence.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if normalize(p) == normalize(g))
    return (correct / len(gold)) * 100

def compute_perplexity(model, testset):
    total_log_prob = 0
    total_tokens = 0
    for doc in testset:
        log_probs = model.log_prob(doc)
        total_log_prob += sum(log_probs)
        total_tokens += len(doc)
    return math.exp(-total_log_prob / total_tokens)

Common pitfalls

• Static benchmarks are prone to data contamination over time; the authors mitigate this for their custom LM testset by enforcing a strict post-training cutoff date (Sept 1, 2023).
• Few-shot settings differ across benchmarks (5-shot for knowledge, 8-shot for GSM8K), requiring consistent prompt formatting to avoid unfair comparisons.
• Perplexity evaluation is restricted to Chinese domains only, making cross-lingual language modeling comparisons impossible.

Evidence (verbatim from paper)

The metrics for CEVAL, CMMLU and MMLU are 5-shot accuracy, while for GSM8K it is 8-shot accuracy. Higher numbers indicate better performance.

Citation

@misc{wei2023skywork,
  title={Skywork: A More Open Bilingual Foundation Model},
  author={Wei et al. (2023)},
  year={2023},
  note={arXiv:2310.19341}
}

• arXiv: 2310.19341