Hazard Eval

hazard-eval

HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environments — Zhou et al. (2024) (arXiv:2401.12975, 2024)

What this evaluates

Evaluates embodied agents' decision-making and planning capabilities in dynamically changing disaster environments (fire, flood, wind). It probes the ability to reason about evolving object states, environmental propagation dynamics, and spatial-temporal trade-offs to successfully rescue valuable items.

Datasets

• HAZARD — total 300; splits: train (75), test (25); repo https://github.com/UMass-Foundation-Model/HAZARD

Metrics

• rescued value rate (Value) (primary) — range: [0, 1]
  • Calculated as the total value of successfully rescued objects divided by the total value of all target objects in the scene. Higher is better.
• averaged rescue step (Step) — range: steps
  • The average number of environment simulation steps taken per episode to complete the rescue task. Lower is better.
• averaged damaged rate (Damage) — range: [0, 1] | percent
  • The average proportion of object/environment integrity lost during the episode due to environmental hazards or failed actions. Lower is better.

Input / output format

Input: Textual prompts containing task instructions, object state history (id, location, distance, temperature/value over time), available actions, and optionally semantic segmentation masks. The prompt is structured to feed visual/historical observations into an LLM via an API.

Output: Textual action commands specifying the next low-level action, e.g., 'go pick up object (87)' or selecting from enumerated options like 'A. go pick up object (64)'.

Scoring recipe

def evaluate_episode(actions, initial_objects, final_objects, target_value):
    rescued_value = sum(obj.value for obj in actions.rescued_objects)
    value_rate = rescued_value / target_value if target_value > 0 else 0
    steps = len(actions.sequence)
    damage = sum(obj.integrity_loss for obj in final_objects) / sum(obj.initial_integrity for obj in initial_objects)
    return value_rate, steps, damage

Common pitfalls

• Perception module failures (e.g., Mask-RCNN missing submerged/obscured objects) are often conflated with reasoning failures, but they are separate evaluation conditions ('Without Perception' vs 'With Perception').
• LLMs may generate reasoning paths that contradict their final action selection; scoring only the final action without checking consistency can mask reasoning flaws.
• Environmental dynamics (fire spread, wind displacement) change state between LLM queries, making static evaluation snapshots or single-step predictions misleading.

Evidence (verbatim from paper)

Table 1: The rescued value rate (Value), averaged rescue step (Step), and averaged damaged rate (Damage) of the proposed LLM pipeline (LLM) and all baseline methods. Without perception denotes the scenario that includes the semantic mask in the input, while the with perception scenario excludes the semantic input and requires the agents to perceive the environment with a perception model.

Citation

@misc{zhou2024hazard,
  title={HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environments},
  author={Zhou et al. (2024)},
  year={2024},
  note={arXiv:2401.12975}
}

• arXiv: 2401.12975