Robocse Eval

robocse-eval

RoboCSE: Robot Common Sense Embedding — Daruna et al. (2019) (arXiv:1903.00412, 2019)

What this evaluates

Evaluates a robot's ability to generalize semantic knowledge by predicting object affordances, locations, and materials in unseen environments and inferring ranks of unseen semantic triples. It probes multi-relational embedding performance for common-sense reasoning in residential robotics.

Datasets

• AI2Thor — total 15000; splits: train (-1), val (-1), test (-1)

Metrics

• MRR (primary) — range: [0, 1]
  • Mean Reciprocal Rank: average of 1/rank for each query, where rank is determined by the number of observations in the ground truth distribution.
• Hits@K — range: [0, 1]
  • Percentage of queries where the correct triple appears in the top K ranked predictions.

Input / output format

Input: Semantic triples (head, relation, tail) from AI2Thor, where the model must predict the missing element given the other two.

Output: Ranked list of candidate triples or entities, ordered by predicted likelihood/rank.

Scoring recipe

def compute_metrics(predictions, ground_truth, K=10):
    reciprocal_ranks = []
    hits_at_k = []
    for pred_list, gt in zip(predictions, ground_truth):
        rank = pred_list.index(gt) + 1
        reciprocal_ranks.append(1.0 / rank)
        hits_at_k.append(1.0 if rank <= K else 0.0)
    mrr = sum(reciprocal_ranks) / len(reciprocal_ranks)
    hits = sum(hits_at_k) / len(hits_at_k)
    return mrr, hits

Common pitfalls

• Negative sampling requires filtering perturbed triples to ensure they are not already in the training set; naive perturbation yields worse results.
• Ground truth ranking is based on observation frequency rather than binary truth values, which differs from standard link prediction benchmarks.
• For environment generalization, splits must be performed at the environment level (not triple level) to prevent data leakage across folds.

Evidence (verbatim from paper)

Error metrics similar to those from the relational embedding community (MRR* and Hits@K*) were calculated using the ground truth rank for comparison. The standard MRR and Hits@K were used to measure the algorithm’s performance, allowing us to assess how frequently the robot was correct on the first attempt.

Citation

@misc{daruna2019robocse,
  title={RoboCSE: Robot Common Sense Embedding},
  author={Daruna et al. (2019)},
  year={2019},
  note={arXiv:1903.00412}
}

• arXiv: 1903.00412