# Arc Eval

> Benchmarks systems on the Abstraction and Reasoning Corpus (ARC) by requiring inference of abstract transformation rules from few input-output grid demonstrations and application to novel test cases, reporting the fraction of tasks solved.

- Skill: `qhjqhj00/arc-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/arc-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/arc-eval/raw
- Safety review: PASS (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML, Model Training & Fine-tuning
- Tags: Arc, Benchmark, Evaluation, Fluid Intelligence, Grid Transformation
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-08-22
- Page: https://skillmd.com/skills/qhjqhj00/arc-eval

---


# arc-eval

> On the Measure of Intelligence — Chollet (2019) (arXiv:1911.01547, 2019)

## What this evaluates

Measures general fluid intelligence and developer-aware generalization by requiring systems to infer abstract transformation rules from few input-output grid demonstrations and apply them to novel test cases. It explicitly avoids measuring task-specific memorization or crystallized knowledge, focusing instead on abstraction, reasoning, and broad generalization under strict prior constraints.

## Datasets

- **Abstraction and Reasoning Corpus (ARC)** — total 1000; splits: train (400), public_eval (400), private_eval (200); repo github.com/fchollet/ARC

## Metrics

- `fraction_of_tasks_solved` **(primary)** — range: [0, 1]
  - Binary success per task: 1 if the system produces the exact correct output grid for all test examples in the task, 0 otherwise. The final score is the fraction of successfully solved tasks in the evaluation set.

## Input / output format

**Input**: A set of demonstration examples, each consisting of an input grid and an output grid (typically 3.3 on average), followed by the input grid of one or more test examples.

**Output**: An output grid constructed from scratch, specifying its dimensions, symbol placement, and layout that correctly transforms the test input grid according to the inferred rule.

## Scoring recipe

```python
score = 0
for task in evaluation_set:
    task_success = True
    for test_input in task.test_examples:
        predicted_output = model.generate_output(task.train_examples, test_input)
        if predicted_output != task.gold_output:
            task_success = False
            break
    if task_success:
        score += 1
final_score = score / len(evaluation_set)
```

## Common pitfalls

- Assuming the output grid dimensions are fixed or provided; the solver must determine height and width from scratch.
- Relying on memorization or pattern matching from the training set, as evaluation tasks are explicitly disjoint and designed to prevent developer-aware shortcuts.
- Ignoring the 3-trial limit per test example; systems must converge to the correct answer within this constraint and only receive binary feedback.

## Evidence (verbatim from paper)

> The task is successfully solved if the test-taker can produce the exact correct answer on all test examples for the task (binary measure of success). For each test example in a task, the test-taker (either human or machine) is allowed 3 trials. The score of an intelligent system on ARC is the fraction of tasks in the evaluation set that it can successfully solve.

## Citation

```bibtex
@misc{chollet2019measure,
  title={On the Measure of Intelligence},
  author={Chollet (2019)},
  year={2019},
  note={arXiv:1911.01547}
}
```

- arXiv: 1911.01547