Robonar Eval

robonar-eval

I Can Tell What I am Doing: Toward Real-World Natural Language Grounding of Robot Experiences — Wang et al. (2024) (arXiv:2411.12960, 2024)

What this evaluates

Evaluates a multimodal robot narration framework's ability to select key events, generate natural language summaries, and perform failure analysis (risk estimation, localization, explanation, recovery) on real-world household robot tasks.

Datasets

• RoboNar — total 70; splits: full (70)

Metrics

• Accuracy on failure analysis tasks (primary) — range: [0, 1]
  • Calculated as the proportion of correctly predicted outcomes across four sub-tasks (Risk Estimation, Failure Localization, Failure Explanation, Recovery Recommendation) out of the total failure cases.
• narration_quality_likert — range: [1, 5]
  • Mean score on a 1-5 Likert scale across four dimensions: Naturalness, Informativeness, Coherence, and Overall Quality, rated by human participants.
• failure_capture_rate — range: [0, 1]
  • Ratio of successfully captured failure cases to the total number of failure cases in the dataset, evaluated across different sampling thresholds and modalities.

Input / output format

Input: Synchronized multimodal sensor streams (RGB-D video from two cameras, joint/base readings, state information, diagnostics) and task planning history, downsampled into aligned keyframes.

Output: Natural language narrations summarizing robot experiences, and structured failure analysis predictions (risk status, failure timestamp, failure reason, recovery recommendation).

Scoring recipe

def score_failure_analysis(predictions, gold):
    correct = 0
    for pred, gold_item in zip(predictions, gold):
        if (pred['risk'] == gold_item['risk'] and
            abs(pred['timestamp'] - gold_item['timestamp']) < 1.0 and
            pred['reason'] == gold_item['reason'] and
            pred['recovery'] == gold_item['recovery']):
            correct += 1
    return correct / len(gold)

def score_narration_quality(ratings):
    return {dim: sum(r[dim]) / len(r) for dim in ['Naturalness', 'Informativeness', 'Coherence', 'Overall']}

Common pitfalls

• Feeding raw, unsummarized multimodal sensor data directly to LLMs/VLMs yields significantly lower accuracy than using intermediate key-event summarization.
• Relying solely on visual inputs for failure analysis is insufficient; internal robot state and planning data are critical for accurate localization and explanation.
• User study latency measurements must be carefully timed per task, as raw data interfaces may appear informative but do not improve accuracy over video-only baselines.

Evidence (verbatim from paper)

We decompose the failurle analysis into four specific tasks: 1) Risk Estimation (Pred): if the method can identify risk before failure happens; 2) Failure Localization (Loc): if the method can identify the failure time ; 3) Failure Explanation (Exp): if the method can tell the failure reason; and 4) Recovery Recommendation (Rec): if the method can give reasonable recovery recommendations. Results are shown in Figure 5: Accuracy on failure analysis tasks using different methods.

Citation

@misc{wang2024robonar,
  title={I Can Tell What I am Doing: Toward Real-World Natural Language Grounding of Robot Experiences},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2411.12960}
}

• arXiv: 2411.12960