Phyre Eval

phyre-eval

PHYRE: A New Benchmark for Physical Reasoning — Bakhtin et al. (2019) (arXiv:1908.05656, 2019)

What this evaluates

Evaluates an agent's ability to reason about 2D Newtonian physics to solve goal-driven puzzles by placing dynamic objects. It probes sample-efficient learning and generalization across unseen task templates and action spaces.

Datasets

• PHYRE — total ?; splits: train (-1), val (-1), test (-1); repo https://phyre.ai

Metrics

• AUCCESSION (primary) — range: percent
  • Area under the success-percentage curve as a function of the number of solution attempts per task (up to 100). Higher values indicate better sample efficiency and overall solving capability.
• success percentage — range: percent
  • The percentage of tasks solved out of the total test tasks at a fixed number of solution attempts per task (e.g., k=10).

Input / output format

Input: A 256x256 image encoding the initial state and goal (7 colors representing body properties and goal), plus task template ID in the within-template setting.

Output: An action from a 3D or 6D action space (depending on the PHYRE tier), representing the placement and properties of a dynamic object to be placed in the simulation.

Scoring recipe

def compute_metrics(predictions, gold, max_attempts=100):
    success_curve = []
    for k in range(1, max_attempts + 1):
        solved = 0
        for task in gold:
            for action in predictions[task][:k]:
                if simulate(task, action).solved:
                    solved += 1
                    break
        success_curve.append(solved / len(gold))
    auc = np.trapz(success_curve) * 100
    return auc

Common pitfalls

• Online agents are permitted up to 100 attempts per task regardless of the evaluation point k, meaning they learn from 100 attempts even when measuring success at k<100.
• Hyperparameter tuning must strictly use the validation set; using test performance for tuning violates the benchmark protocol.
• Statistical significance is evaluated using the non-parametric Wilcoxon signed-rank test, not standard t-tests.

Evidence (verbatim from paper)

We measure success percentage and AUCCESSION on PHYRE using the learning setting of 3.2. ... Figure 3 presents success-percentage curves for all five agents on both PHYRE tiers (-B and -2B) in both generalization settings (within-template and cross-template): the curves show the percentage of tasks solved as a function of the number of solution attempts per task, and are computed by averaging over all 10 folds in PHYRE.

Citation

@misc{bakhtin2019phyre,
  title={PHYRE: A New Benchmark for Physical Reasoning},
  author={Bakhtin et al. (2019)},
  year={2019},
  note={arXiv:1908.05656}
}

• arXiv: 1908.05656