Catwalk Default Metrics

catwalk-default-metrics

Catwalk: A Unified Language Model Evaluation Framework for Many Datasets — Groeneveld et al. (2023) (arXiv:2312.10253, 2023)

What this evaluates

Evaluates language models across multiple dataset categories using standardized metrics attached to datasets rather than model implementations. Probes classification/entailment accuracy, question-answering fidelity, and language modeling fluency/probability calibration.

Datasets

• (no dataset; pure metric skill)

Metrics

• Accuracy & relative improvement over random baseline (primary) — range: [0, 1] | percent
  • Computes standard accuracy for multiple-choice, classification, and entailment tasks. Also calculates relative improvement over a uniform random baseline.
• SQuAD metric (primary) — range: [0, 1]
  • Follows the exact implementation by Rajpurkar et al. (2016), computing token-level exact match and F1 score for open-ended question answering.
• Perplexity per word, perplexity per byte, and entropy — range: other
  • Computes average log-probability per token (perplexity per word), per byte of text, and Shannon entropy over the predicted token distribution for language modeling tasks.

Input / output format

Input: Dataset-specific inputs: multiple-choice prompts, classification/entailment pairs, open-ended QA questions, or raw text sequences for language modeling.

Output: Model predictions including class labels, generated text spans, or token-level probability distributions.

Scoring recipe

def compute_metrics(predictions, gold, dataset_type, model):
    if dataset_type in ['multiple-choice', 'classification', 'entailment']:
        acc = mean(predictions == gold)
        random_acc = 1.0 / num_classes
        return {'accuracy': acc, 'relative_improvement': (acc - random_acc) / random_acc}
    elif dataset_type == 'qa':
        return squad_metric(predictions, gold) # Rajpurkar et al. 2016
    elif dataset_type == 'lm':
        log_probs = model.log_prob(predictions)
        return {
            'perplexity_per_word': exp(-mean(log_probs)),
            'perplexity_per_byte': exp(-mean(log_probs) / mean(len(token) for token in predictions)),
            'entropy': -mean(sum(p * log(p) for p in dist))
        }

Common pitfalls

• Metrics are attached to datasets, not models; comparing models requires ensuring the same dataset-attached metrics are used.
• The Eleuther model style does not automatically compute suggested metrics, unlike other implemented styles.
• SQuAD metric refers to the specific Rajpurkar et al. (2016) implementation, which computes both exact match and F1, not just a single score.

Evidence (verbatim from paper)

By default, Catwalk computes the following metrics and can be easily extended to include others. - For multiple-choice, classification, and entailment datasets: Accuracy and relative improvement over the random baseline - For question-answering datasets: SQuAD metric as defined by Rajpurkar et al. (2016). - For language modeling datasets: Perplexity per word, perplexity per byte, and entropy.

Citation

@misc{groeneveld2023catwalk,
  title={Catwalk: A Unified Language Model Evaluation Framework for Many Datasets},
  author={Groeneveld et al. (2023)},
  year={2023},
  note={arXiv:2312.10253}
}

• arXiv: 2312.10253