Glm 130b Eval

glm-130b-eval

GLM-130B: An Open Bilingual Pre-trained Model — Aohan Zeng et al. (arXiv:2210.02414, 2022)

What this evaluates

This evaluation protocol assesses the zero-shot and few-shot capabilities of large bilingual language models across diverse English and Chinese benchmarks. It probes language modeling, multi-choice question answering, reasoning, commonsense, and cross-lingual transfer abilities.

Datasets

• LAMBADA — total ?; splits: test (-1)
• Pile — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)
• BIG-bench-lite — total ?; splits: test (-1)
• CLUE — total ?; splits: dev (-1)
• FewCLUE — total ?; splits: dev (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted labels or next words compared to ground truth. Calculated as the mean of binary matches between normalized predictions and gold answers across all instances.
• BPB — range: other
  • Bits Per Byte, computed as the negative log-likelihood of the test set normalized by the total number of bytes. Used for language modeling evaluation on the Pile.

Input / output format

Input: Text prompts formatted for zero-shot or few-shot evaluation. For multiple-choice tasks, the prompt includes the question and options, optionally preceded by in-context examples. For language modeling, the prompt is a text sequence ending with a blank or partial sentence.

Output: Predicted next token(s) or selected option label. For multiple-choice QA, the model outputs the chosen answer string or letter. For language modeling, it generates the continuation.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if normalize(p) == normalize(g))
    return correct / len(gold_labels)

def compute_bpb(log_probs, byte_counts):
    total_bits = sum(log_probs * byte_counts)
    return total_bits / sum(byte_counts)

Common pitfalls

• Data contamination: Datasets must be filtered for 13-gram overlap with training corpora and excluded if released after training data crawling.
• Zero-shot definition: The paper uses a specific criterion involving unseen class labels or obvious domain transfer, which may differ from standard community usage.
• Few-shot bias: Models trained with MIP are biased toward zero-shot performance and may show weaker few-shot scaling compared to autoregressive models.

Evidence (verbatim from paper)

LAMBADA (Paperno et al., 2016) is a dataset to test the last word language modeling capability. The results previously shown in Figure 2 suggest GLM-130B achieves a zero-shot accuracy of 80.2 with its bidirectional attention, setting up a new record on LAMBADA.

Citation

@misc{zeng2022glm130b,
  title={GLM-130B: An Open Bilingual Pre-trained Model},
  author={Aohan Zeng et al.},
  year={2022},
  note={arXiv:2210.02414}
}

• arXiv: 2210.02414