Chaic Eval

chaic-eval

Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challenge — Du et al. (2024) (arXiv:2411.01796, 2024)

What this evaluates

Evaluates AI agents' ability to socially perceive and cooperatively assist physically constrained humans in long-horizon indoor and outdoor tasks. It probes cooperative planning, goal inference from egocentric visual input, and emergency response under physical constraints.

Datasets

• CHAIC — total ?; splits: test (-1); repo https://github.com/UMass-Embodied-AGI/CHAIC

Metrics

• Transport Rate (TR) (primary) — range: [0, 1]
  • The percentage of target objects that the agents successfully transported. Calculated as (number of successfully transported target objects) / (total target objects).
• Efficiency Improvement (EI) — range: [0, 1]
  • ΔM/M0, where ΔM denotes the increase in the transport rate after adding the helper, and M0 denotes the larger of the transport rates of the team or the constrained agent alone, for numerical stability.
• Goal Inference Accuracy (IA) — range: [0, 1]
  • The ratio of target objects successfully transported by the helper to the total number of objects transported by the helper.
• Emergency Rate (ER) — range: [0, 1]
  • For the shopping task only, the ratio of frames where the child agent is away from the constrained agent to measure the helper agent’s ability to handle emergencies. Lower is better.

Input / output format

Input: Egocentric RGB-D video frames (last 10 frames for VLM baselines) and text descriptions of object locations/goals. Constrained agent uses ground truth object segmentation.

Output: High-level cooperative plans or discrete actions (e.g., 'goto and pick up target ', 'transport object in hand to goal space', 'put object in container').

Scoring recipe

def compute_metrics(gold_targets, helper_actions, constrained_alone_tr, team_alone_tr, frames_away=None, total_frames=None):
    transported_targets = count_transported(gold_targets, helper_actions)
    total_transported = count_all_transported(helper_actions)
    TR = transported_targets / len(gold_targets)
    M0 = max(constrained_alone_tr, team_alone_tr)
    EI = (TR - M0) / M0 if M0 > 0 else 0.0
    IA = transported_targets / total_transported if total_transported > 0 else 0.0
    ER = frames_away / total_frames if frames_away is not None else None
    return TR, EI, IA, ER

Common pitfalls

• LLM/VLM helpers often fail to understand spatial relationships from text inputs, choosing distant objects over nearby ones and underestimating travel costs.
• VLM baselines can perform worse than random or no-helper baselines because they blindly follow the constrained agent, blocking paths or failing to transport any objects.
• Emergency Rate (ER) is only calculated for the outdoor shopping task, not across all constraint settings.

Evidence (verbatim from paper)

To evaluate the success of helper agents, we measure the following three metrics: Transport rate (TR): The percentage of target objects that the agents successfully transported. We also calculate the Efficiency Improvement (EI) of having the helper as ΔM/M0, where ΔM denotes the increase in the transport rate after adding the helper, and M0 denotes the larger of the transport rates of the team or the constrained agent alone, for numerical stability. Goal Inference Accuracy (IA): The ratio of target objects successfully transported by the helper to the total number of objects transported by the helper. Emergency Rate (ER): For the shopping task, we calculate the ratio of frames where the child agent is away from the constrained agent to measure the helper agent’s ability to handle emergencies.

Citation

@misc{du2024chaic,
  title={Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challenge},
  author={Du et al. (2024)},
  year={2024},
  note={arXiv:2411.01796}
}

• arXiv: 2411.01796