Coin Bench Eval

coin-bench-eval

Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialogues — Taioli et al. (2024) (arXiv:2412.01250, 2024)

What this evaluates

Evaluates an agent's ability to navigate to a specific target instance in multi-instance scenes through collaborative, open-ended dialogues with a human or simulated user. It probes the agent's uncertainty-aware reasoning, dialogue efficiency, and generalization to unseen object categories.

Datasets

• CoIN-Bench — total ?; splits: Val Seen (-1), Val Seen Synonyms (-1), Val Unseen (-1), Train (-1)
• IDKVQA — total 502; splits: test (502)

Metrics

• SR (primary) — range: percent
  • Success Rate: percentage of navigation episodes where the agent successfully reaches the target instance within the allowed steps and dialogue rounds.
• SPL — range: [0, 1]
  • Success weighted by Path Length: standard navigation metric combining success rate and path efficiency relative to the optimal path.
• NQ — range: other
  • Number of Questions: average count of questions asked by the agent to the user per episode to resolve ambiguity.
• $\Phi_{c=1}$ — range: percent
  • Effective Reliability metric at confidence threshold c=1: measures the proportion of correctly identified uncertain predictions by the VLM uncertainty estimation technique.

Input / output format

Input: Visual observations from the agent's viewpoint, target instance description or category, and dialogue history. For human evaluation, users receive an image of the target instance and interact via a chat interface.

Output: Navigation actions (e.g., move forward, turn, stop) and natural language questions directed to the user. Maximum 4 dialogue rounds allowed.

Scoring recipe

def compute_metrics(predictions, gold):
    sr = sum(1 for p, g in zip(predictions, gold) if p.success) / len(predictions) * 100
    spl = sum(p.spl * (1 if p.success else 0) for p in predictions) / len(predictions)
    nq = sum(p.num_questions for p in predictions) / len(predictions)
    return sr, spl, nq

def compute_phi_c1(predictions, gold, threshold=1.0):
    correct_uncertain = sum(1 for p, g in zip(predictions, gold) if p.uncertain and not p.correct)
    total_uncertain = sum(1 for p in predictions if p.uncertain)
    return (correct_uncertain / total_uncertain) * 100 if total_uncertain > 0 else 0.0

Common pitfalls

• Simulation setup is validated on only 40 episodes; results may not fully generalize to real human cognitive load or interaction patterns.
• Baselines differ in input granularity (category vs. detailed description), making direct SR comparisons potentially unfair without accounting for instance-level discrimination capabilities.
• The uncertainty threshold $\tau$ is highly sensitive to dataset size and requires careful tuning per split.

Evidence (verbatim from paper)

Tab.[5] reports the performance using the Effective Reliability metric $\Phi_{c}$ proposed in[[50]]. Our proposed technique achieves the best $\Phi_{c=1}$ score of 21.12, demonstrating its effectiveness.

Citation

@misc{taioli2024collaborative,
  title={Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialogues},
  author={Taioli et al. (2024)},
  year={2024},
  note={arXiv:2412.01250}
}

• arXiv: 2412.01250