Big Bench Eval

big-bench-eval

ThinkSum: Probabilistic reasoning over sets using large language models — Ozturkler et al. (2022) (arXiv:2210.01293, 2022)

What this evaluates

Evaluates large language models on probabilistic reasoning over sets, including conceptual combination, semantic outlier detection, and logical deduction. It probes the model's ability to decouple knowledge retrieval from final inference and aggregate uncertain facts without relying solely on self-attention.

Datasets

• BIG-bench — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted answers across all instances in a task. Calculated as the number of matches between the model's selected option and the ground truth label divided by the total number of instances.
• BLEU — range: [0, 1]
  • Standard BLEU score used specifically for the CODENAMES task to evaluate the quality of generated text against reference translations.

Input / output format

Input: Task-specific prompts containing instructions, few-shot examples (where applicable), and problem instances (e.g., definitions of nonce words, lists of items to compare, or ordering clues).

Output: Model-generated completions, typically selecting a single option from a provided set (e.g., a statement, a word, or a logical placement).

Scoring recipe

def compute_metrics(predictions, golds):
    acc_scores = []
    for pred, gold in zip(predictions, golds):
        acc_scores.append(1.0 if pred == gold else 0.0)
    return sum(acc_scores) / len(acc_scores)

Common pitfalls

• Prompt length can degrade performance for auxiliary knowledge baselines, peaking and then dropping as context grows.
• Smaller models (e.g., GPT-2 Small) may fail to reliably execute complex 'Think' prompts, leading to missing evaluations.
• API quotas and dataset sizes may prevent evaluation of certain model-task combinations (e.g., InstructGPT on LANGUAGE IDENTIFICATION).

Evidence (verbatim from paper)

Table 1: Standard metric (BLEU for CODENAMES, accuracy for other tasks) for GPT-3 175B (davinci) and ThinkSum with 175B (davinci), InstructGPT and GPT-2 XL on BIG-bench tasks.

Citation

@misc{ozturkler2022thinksum,
  title={ThinkSum: Probabilistic reasoning over sets using large language models},
  author={Ozturkler et al. (2022)},
  year={2022},
  note={arXiv:2210.01293}
}

• arXiv: 2210.01293