qhjqhj00 household-rearrangement-eval
LLM-enhanced Scene Graph Learning for Household Rearrangement — Wenhao Li et al. (arXiv:2408.12093, 2024)
What this evaluates
Evaluates an agent's ability to detect misplacements of objects in indoor scenes and plan optimal rearrangement placements for carryable objects based on scene context and affordances. It probes commonsense reasoning about object-receptacle relationships and ranking quality under varying contextual cues.
Datasets
- Tidybot benchmark — total ?; splits: test (-1)
- Context-oriented benchmark (HSSD 200) — total 4000; splits: test (-1)
Metrics
NDCG@8(primary) — range: [0, 1]- Normalized Discounted Cumulative Gain calculated over the top 8 recommended receptacle placements. Measures how closely the predicted ranking matches the human-annotated ground truth ranking.
Accuracy/Recall/Precision/F1— range: [0, 1]- Standard classification metrics for misplacement detection. Accuracy is the fraction of correctly classified objects; Recall, Precision, and F1 are computed from true/false positives and negatives.
Input / output format
Input: Indoor scene representation (scene graph enhanced with visual, textual, and global context), target carryable object, and list of available receptacles.
Output: Ranked list of recommended receptacle placements for the target object; binary/multi-class classification output for misplacement detection.
Scoring recipe
def compute_ndcg_at_k(pred_rank, gt_rank, k=8):
dcg = sum(r / math.log2(i + 2) for i, r in enumerate(gt_rank[:k]))
idcg = sum(r / math.log2(i + 2) for i, r in enumerate(sorted(gt_rank, reverse=True)[:k]))
return dcg / idcg if idcg > 0 else 0.0
def compute_detection_metrics(preds, golds):
tp = sum(p == g == 1 for p, g in zip(preds, golds))
fp = sum(p == 1 and g == 0 for p, g in zip(preds, golds))
fn = sum(p == 0 and g == 1 for p, g in zip(preds, golds))
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
acc = sum(p == g for p, g in zip(preds, golds)) / len(preds)
return acc, rec, prec, f1
Common pitfalls
- NDCG is strictly calculated on the top 8 placements, not the full candidate list.
- Ground truth rankings are derived from majority votes of human-annotated robot trajectories, not single placements.
- Tidybot baseline requires imitation demonstrations for planning, whereas the proposed method uses them only as sparse context, creating an asymmetric evaluation setting.
Evidence (verbatim from paper)
Metrics. For the household rearrangement task, we evaluated our method mainly in two aspects: misplacement detection and carriable object rearrangement planning. When the ground-truth object placements are available, we measure the detection performance based on accuracy, recall, precision, and F1 score following most existing works [Padilla et al. 2020]. In our benchmark tests, each object may have multiple placement positions, with a ranking that reflects human common sense. To more comprehensively compare the performance of different methods in carriable object rearrangement planning, we also require these methods to output a ranking of recommended placement positions. Normalized Documented Cumulative Gain, which is regarded as a measure of how close a ranking is to the ground truth ranking, can assess whether the methods can select appropriate placement positions based on human common sense in the rearrangement planning. In practice, we take the best 8 placements for each carriable object to calculate the NDCG.
Citation
@misc{li2024llmenhanced,
title={LLM-enhanced Scene Graph Learning for Household Rearrangement},
author={Wenhao Li et al.},
year={2024},
note={arXiv:2408.12093}
}
- arXiv: 2408.12093