Crew Wildfire Eval

crew-wildfire-eval

CREW-WILDFIRE: Benchmarking Agentic Multi-Agent Collaborations at Scale — Hyun et al. (2025) (arXiv:2507.05178, 2025)

What this evaluates

Probes LLM-based multi-agent coordination in dynamic, partially observable wildfire disaster response scenarios. It evaluates capabilities such as spatial reasoning, task designation, plan adaptation, and heterogeneous team collaboration under stochastic dynamics and long-horizon objectives.

Datasets

• CREW-Wildfire — total ?; splits: test (-1)

Metrics

• task success (primary) — range: [0, 1]
  • Binary or proportional measure of whether all level-specific objectives are completed (e.g., all trees cut, all civilians rescued, fire extinguished).
• Composite Score (Trees Destroyed + 20×Agents Lost + 100×Civilians Lost) — range: other
  • Level-specific scoring functions aggregate task completion, damage minimization, and safety penalties. For the Full Environment level, it is explicitly Trees Destroyed + 20×Agents Lost + 100×Civilians Lost.

Input / output format

Input: Procedurally generated wildfire environment state including map dimensions, agent types (Firefighters, Bulldozers, Drones, Helicopters), civilian and fire locations, partial observability constraints, and level-specific objectives.

Output: Agent action sequences or coordination plans executed over a long-horizon simulation, evaluated against level-specific scoring functions and behavioral competency markers.

Scoring recipe

def compute_score(level, final_state):
    if level == 'Full Environment':
        return final_state.trees_destroyed + 20 * final_state.agents_lost + 100 * final_state.civilians_lost
    elif level == 'Cut Trees':
        return final_state.trees_cut_in_labeled_cells
    elif level == 'Scout Fire':
        return min(2, final_state.drones_over_fire)
    elif level in ['Transport', 'Rescue']:
        return final_state.units_at_target
    else:
        return final_state.trees_destroyed + 20 * final_state.agents_lost

Common pitfalls

• Assuming a single fixed scoring formula applies across all 12 levels; each level uses a distinct scoring function (e.g., trees cut vs. drones over fire vs. composite penalties).
• Overlooking partial observability and stochastic fire spread, which require agents to adapt plans and share observations rather than relying on global state.
• Treating behavioral competencies as optional; they are explicitly marked per level to diagnose specific coordination failures.

Evidence (verbatim from paper)

We measure the performance via task success, damage minimization, and agent/civilian safety. We also assign high-level behavioral goals to each level to evaluate a variety of high-level behaviors. The behavioral goals include: 1) Task Designation (TD): the ability to divide tasks among agents; 2) Agent Capitalization (AC): the ability to recognize and capitalize on the strengths and weaknesses in heterogeneous teams; 3) Spatial Reasoning (SR): The ability to reason and plan accordingly with spatial information; 4) Observation Sharing (OS): the ability to communicate useful observations when necessary; 5) Realtime Coordination (RC): the ability to communicate and rely on other agents to perform synchronized tasks; 6) Plan Adaptation (PA): the ability to adapt and revise plans; 7) Objective Prioritization (OP): the ability to rank competing goals contextually. ... Full Environment Scoring Function: Trees Destroyed + 20×Agents Lost + 100×Civilians Lost

Citation

@misc{hyun2025crewwildfire,
  title={CREW-WILDFIRE: Benchmarking Agentic Multi-Agent Collaborations at Scale},
  author={Hyun et al. (2025)},
  year={2025},
  note={arXiv:2507.05178}
}

• arXiv: 2507.05178