# Bbh Eval

> Benchmarks zero-shot in-context learning on BIG-Bench Hard multiple-choice tasks, comparing self-generated demonstrations against direct prompting and chain-of-thought baselines, and reports accuracy.

- Skill: `qhjqhj00/bbh-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/bbh-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/bbh-eval/raw
- Safety review: PASS (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML, Model Training & Fine-tuning
- Tags: Accuracy, Bbh, Big Bench Hard, Chain Of Thought, Evaluation, In Context Learning, Zero Shot
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-08-22
- Page: https://skillmd.com/skills/qhjqhj00/bbh-eval

---


# bbh-eval

> Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations — Chen et al. (2023) (arXiv:2305.15035, 2023)

## What this evaluates

Evaluates a model's zero-shot in-context learning capability on reasoning-heavy multiple-choice tasks. It compares self-generated demonstrations against direct prompting and chain-of-thought baselines to measure accuracy gains.

## Datasets

- **BIG-Bench Hard (BBH)** — total 5511; splits: test (5511)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard exact-match accuracy for multiple-choice tasks: the proportion of test instances where the model's predicted answer exactly matches the gold label.

## Input / output format

**Input**: Task description followed by the test input instance. For CoT baselines, a reasoning trigger phrase 'Let's think step by step.' is appended.

**Output**: A single predicted answer choice from the multiple-choice options.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().lower() == gold.strip().lower():
        correct += 1
return correct / len(gold_labels)
```

## Common pitfalls

- BBH contains 27 tasks total, but the evaluation strictly uses only the 23 multiple-choice tasks; including non-multiple-choice tasks will break the accuracy metric.
- Evaluation temperature must be set to 0 for deterministic results, as specified in the implementation details.
- Prompt format varies significantly between baselines (ZS-Direct vs ZS-CoT); mixing prompt templates will invalidate head-to-head comparisons.

## Evidence (verbatim from paper)

> We adopt the BIG-Bench Hard (BBH) benchmark for our evaluation. BBH contains a total of 27 tasks, from which we select 23 tasks that are multiple-choice tasks as our evaluation testbed for SELF-ICL. Each BBH tasks has around 150 ~ 250 examples, and the total number of instances is 5,511. The accuracy delta indicates the accuracy difference between SELF-ICL and the baseline method (blue/orange indicates our method wins/loses).

## Citation

```bibtex
@misc{chen2023selficl,
  title={Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations},
  author={Chen et al. (2023)},
  year={2023},
  note={arXiv:2305.15035}
}
```

- arXiv: 2305.15035