qhjqhj00 deepfurniture-eval
Furnishing Your Room by What You See: An End-to-End Furniture Set Retrieval Framework with Rich Annotated Benchmark Dataset — Liu et al. (2019) (arXiv:1911.09299, 2019)
What this evaluates
Evaluates furniture detection, segmentation, instance retrieval, and set retrieval in indoor scenes. It probes occlusion robustness, fine-grained attribute-based feature learning, and spatial co-occurrence modeling for interior design understanding.
Datasets
- DeepFurniture — total 24000; splits: train (19000), test (5000)
Metrics
AP(primary) — range: percent- Average Precision computed over IoU thresholds (0.50 and 0.75) for both bounding boxes and segmentation masks, following COCO evaluation protocol.
ACC@K(primary) — range: percent- Accuracy at rank K: the fraction of queries where the ground-truth instance appears within the top-K retrieved results.
SET ACC@K— range: percent- Set Accuracy at rank K: the fraction of test images where all ground-truth furniture instances in the scene appear within the top-K retrieved instances.
Input / output format
Input: Indoor images for detection; cropped furniture instances or image patches for instance retrieval; indoor images containing multiple furniture items for set retrieval.
Output: Bounding boxes and segmentation masks for detection; a ranked list of furniture instance IDs for retrieval; a ranked list of furniture sets for set retrieval.
Scoring recipe
def compute_acc_at_k(retrieved_ids, gt_id, k):
return 1.0 if gt_id in retrieved_ids[:k] else 0.0
def compute_set_acc_at_k(retrieved_instances, gt_set, k):
return 1.0 if gt_set.issubset(retrieved_instances[:k]) else 0.0
def compute_ap(preds, gts, iou_thresh=0.5):
# Standard COCO AP calculation over IoU thresholds
...
Common pitfalls
- Set retrieval accuracy is inherently low because errors in the detection and instance retrieval stages amplify, making the final set-level metric highly sensitive to minor mistakes.
- Using category-level supervision for feature learning can actually degrade instance retrieval performance compared to a pre-trained baseline, as broad categories lack fine-grained visual distinction.
Evidence (verbatim from paper)
The top-1 and top-5 instance accuracy of our system is 45.4% and 57.4% respectively, while the top-5 and top-10 set accuracy is 8.60% and 11.3% respectively.
Citation
@misc{liu2019furnishing,
title={Furnishing Your Room by What You See: An End-to-End Furniture Set Retrieval Framework with Rich Annotated Benchmark Dataset},
author={Liu et al. (2019)},
year={2019},
note={arXiv:1911.09299}
}
- arXiv: 1911.09299