Lingo Space Grounding Eval

lingo-space-grounding-eval

LINGO-Space: Language-Conditioned Incremental Grounding for Space — Kim et al. (2024) (arXiv:2402.01183, 2024)

What this evaluates

Evaluates a model's ability to ground natural language spatial instructions to specific 2D pixel locations in RGB-D tabletop scenes. It probes both single-relation grounding and incremental/compositional grounding where multiple spatial predicates must be satisfied sequentially or simultaneously.

Datasets

• CLIPort Benchmark — total 300; splits: train (100), test (200)
• ParaGon Benchmark — total 600; splits: train (400), test (200)
• SREM Benchmark — total 300; splits: train (100), test (200)
• LINGO-Space Benchmark — total 300; splits: train (100), test (200)
• Composite Instruction Task — total 500; splits: train (200), test (300)

Metrics

• success score (primary) — range: [0, 1]
  • A continuous score in [0, 1] reflecting the extent of relationship satisfaction between the located object volume and the desired container/reference objects based on spatial predicates.
• binary success — range: {0, 1}
  • A binary score in {0, 1} indicating whether all spatial predicates in the instruction are strictly satisfied after placement.
• satisfaction ratio — range: [0, 1]
  • For composite instructions: score = #satisfied_relations / |total_relations|.

Input / output format

Input: Top-down RGB-D images (640x320) of tabletop scenes containing 4-7 objects, paired with synthesized natural language instructions containing one or multiple spatial referring expressions (e.g., 'left', 'right', 'close', 'far').

Output: A 2D pixel coordinate or bounding box indicating the target placement location for the referenced object.

Scoring recipe

def score(predictions, gold, relations):
    satisfied = 0
    for rel in relations:
        if check_spatial_relation(predictions, gold, rel):
            satisfied += 1
    if len(relations) == 1:
        return extent_of_satisfaction(predictions, gold) # [0,1]
    else:
        return satisfied / len(relations) # [0,1]
# Binary success is 1 if all relations satisfied, else 0.

Common pitfalls

• Spatial ambiguity arises when multiple semantically identical objects are present, requiring precise disambiguation via incremental grounding.
• Defining strict thresholds for continuous spatial predicates like 'close' vs 'far' or directional relations can lead to inconsistent scoring if not standardized.
• Compositional instructions require maintaining state across multiple relations; failing to update the spatial distribution after each step breaks subsequent grounding.

Evidence (verbatim from paper)

The assessment metric is a success score ($\in[0,1]$) reflecting the extent of relationship satisfaction between the located object volume and the desired container. For composite tasks, we compute a score reflecting the extent of relationship satisfaction; $score=#satisfaction/|relations|$.

Citation

@misc{kim2024lingospace,
  title={LINGO-Space: Language-Conditioned Incremental Grounding for Space},
  author={Kim et al. (2024)},
  year={2024},
  note={arXiv:2402.01183}
}

• arXiv: 2402.01183