Scienceworld Eval

scienceworld-eval

ScienceWorld: Is your Agent Smarter than a 5th Grader? — Wang et al. (2022) (arXiv:2203.07540, 2022)

What this evaluates

Evaluates an agent's ability to perform procedural scientific reasoning and navigation within an interactive text-based environment. It probes whether models can execute multi-step experiments (e.g., building circuits, measuring temperatures) rather than just retrieving static facts.

Datasets

• ScienceWorld — total 7200; splits: train (3600), dev (1800), test (1800); repo https://github.com/allenai/ScienceWorld

Metrics

• average_score (primary) — range: [0, 1]
  • The mean normalized task score across all 30 subtasks. Each task's score is normalized to [0, 1] based on the completion of required and optional sub-goals during the episode.

Input / output format

Input: Text observation of the current state (room description, inventory), task description, and optionally previous observation/action.

Output: A text command (action) selected from the set of valid simulator actions.

Scoring recipe

def evaluate(agent, tasks):
    total_score = 0.0
    for task in tasks:
        task_scores = []
        for variation in task.test_variations:
            # Environment runs agent, tracks required/optional subgoals
            score = env.run(agent, variation) 
            task_scores.append(score)
        total_score += mean(task_scores)
    return total_score / len(tasks)

Common pitfalls

• Models may solve tasks via knowledge retrieval rather than procedural execution; the benchmark includes paired tasks (known vs. unknown entities) to distinguish these capabilities.
• RL agents' performance is averaged over the last 10% of evaluation episodes, while language model agents are averaged across all test variations, making direct comparison sensitive to evaluation windows.
• Agents generating invalid actions receive zero scores unless a valid-action detection aid is used, which significantly impacts reported performance.

Evidence (verbatim from paper)

Scores for all tasks are normalized to between 0 and 1. Meeting required and optional subgoals increases the agent's score on a given subtask. ... Overall, these tasks are challenging for current models, with the best model (DRRN) achieving an average score of 0.17 across all 30 subtasks.

Citation

@misc{wang2022scienceworld,
  title={ScienceWorld: Is your Agent Smarter than a 5th Grader?},
  author={Wang et al. (2022)},
  year={2022},
  note={arXiv:2203.07540}
}

• arXiv: 2203.07540