Unclearinstruct Eval

unclearinstruct-eval

Pragmatic Embodied Spoken Instruction Following in Human-Robot Collaboration with Theory of Mind — Ying et al. (2024) (arXiv:2409.10849, 2024)

What this evaluates

Evaluates a robot's ability to infer human goals and execute household tasks from noisy, accented, or mispronounced spoken instructions. It probes robust speech perception, joint planning, and Theory of Mind in embodied human-robot collaboration under mixed-observability conditions.

Datasets

• UnclearInstruct — total 3080; splits: test (3080)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Percentage of trials where the robot correctly infers the intended robot goal from the speech instruction.
• Speedup — range: [0, 1]
  • Ratio of single-agent task completion timesteps to team completion timesteps, minus one: L_single / L_team - 1.
• Word Error Rate (WER) — range: [0, 1]
  • Percentage of word errors in the generated transcript relative to the total words spoken. A perfect transcription yields 0.

Input / output format

Input: Noisy speech audio file (original, noise-injected, accented, or mispronounced), optionally accompanied by text descriptions of visual observations and the goal space.

Output: A symbolic robot goal specification (e.g., item types and quantities to pick up) and/or a text transcription of the speech instruction.

Scoring recipe

def compute_metrics(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p['goal'] == g['goal'])
    accuracy = correct / len(gold)
    speedups = [(g['timesteps_single'] / p['timesteps']) - 1 for p, g in zip(predictions, gold)]
    avg_speedup = sum(speedups) / len(speedups)
    wer_sum = sum(edit_distance(p['transcript'], g['transcript']) / len(g['transcript']) for p, g in zip(predictions, gold))
    wer = wer_sum / len(gold)
    return {'accuracy': accuracy, 'speedup': avg_speedup, 'wer': wer}

Common pitfalls

• The speech conditions (noise, accents, mispronunciations) are synthetically generated using external tools rather than naturally recorded, limiting real-world acoustic generalization.
• Speedup is tightly coupled to the VirtualHome simulator's timestep definition and baseline comparison (single human vs. human-robot team), making cross-platform comparison difficult.
• WER for SIFToM is computed on Whisper transcripts rather than end-to-end model outputs, which may inflate comparability with pure ASR baselines.

Evidence (verbatim from paper)

We evaluated the model performance with three metrics: accuracy rate, speedup, and word error rare (WER). a) Accuracy: We measured inference accuracy by the percentage of trials that the robot inferred the correct robot goal from speech instructions. b) Speedup: Following past work on human-robot collaboration in VirtualHome [35], we computed the speedup of the baseline models against a single-agent (human alone) baseline. The speedup is computed as $L_{\text{single}} / L_{\text{team}} - 1$, where $L_{\text{single}}$ is the timesteps it takes for the human agent to complete the task alone without robot assistance and $L_{\text{team}}$ is the timesteps for the human-robot team. c) Word Error Rate: We followed existing work on ASR, which calculated WER as the percentage of errors in a transcript compared to the total words spoken. A perfect transcription has a WER of 0.

Citation

@misc{ying2024pragmatic,
  title={Pragmatic Embodied Spoken Instruction Following in Human-Robot Collaboration with Theory of Mind},
  author={Ying et al. (2024)},
  year={2024},
  note={arXiv:2409.10849}
}

• arXiv: 2409.10849