Scene Smith Eval

scene-smith-eval

SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes — Pfaff et al. (2026) (arXiv:2602.09153, 2026)

What this evaluates

Evaluates text-to-3D indoor scene generation systems on their ability to produce dense, physically plausible, and prompt-faithful environments. It probes both visual realism and simulation-readiness, measuring collision-free layouts and stable physics properties required for robotics policy testing.

Datasets

• SceneSmith Prompt Corpus — total 210; splits: test (210)

Metrics

• Realism Win% (primary) — range: percent
  • Percentage of pairwise human comparisons where the generated scene is preferred over the baseline for visual realism. Ties are excluded from the denominator.
• Faithfulness Win% — range: percent
  • Percentage of pairwise human comparisons where the generated scene better follows the text prompt requirements compared to the baseline.
• CNT — range: percent
  • VLM-based score measuring object count accuracy against the prompt.
• ATR — range: percent
  • VLM-based score measuring object attribute correctness.
• OOR — range: percent
  • VLM-based score measuring object-object relationship accuracy.
• OAR — range: percent
  • VLM-based score measuring object-architecture relationship accuracy.
• ACC — range: percent
  • VLM-based score measuring object accessibility.
• NAV — range: percent
  • VLM-based score measuring scene navigability.
• OOB — range: percent
  • VLM-based score measuring out-of-bounds violations (lower is better).
• COL — range: percent
  • Physics simulation collision rate measured using Drake (lower is better).
• STB — range: percent
  • Static equilibrium stability score measured using Drake (higher is better).

Input / output format

Input: Natural language text prompts describing indoor scenes (room-level or house-level), optionally paired with baseline scene assets for direct comparison.

Output: 3D scene representations including asset placements, layouts, collision geometry, and physical properties, formatted for direct import into physics simulators (e.g., Drake).

Scoring recipe

def evaluate_scene(scene, prompt, baseline_scene=None):
    # Human pairwise comparison (if applicable)
    realism_win = human_prefers(scene, baseline_scene, criterion="realism")
    faith_win = human_prefers(scene, baseline_scene, criterion="prompt_following")

    # Automated VLM metrics
    metrics = {
        "CNT": vlm_score(scene, "object_count"),
        "ATR": vlm_score(scene, "attributes"),
        "OOR": vlm_score(scene, "object_relations"),
        "OAR": vlm_score(scene, "arch_relations"),
        "ACC": vlm_score(scene, "accessibility"),
        "NAV": vlm_score(scene, "navigability"),
        "OOB": vlm_score(scene, "out_of_bounds")
    }

    # Physics metrics (requires augmented collision/physics data)
    metrics["COL"] = drake_collision_rate(scene)
    metrics["STB"] = drake_static_equilibrium(scene)

    return {"human": {"realism": realism_win, "faithfulness": faith_win}, "auto": metrics}

Common pitfalls

• VLM-based metrics (CNT, ATR, etc.) are prone to false positives/negatives and require careful validation or manual spot-checking.
• Baseline methods lack native simulation-ready outputs, so collision geometry and physical properties must be artificially augmented to compute COL and STB fairly.
• Human study statistical power is limited; ablations like NoCritic and NoAgentMemory showed non-significant effects that would require 6-18x more comparisons to detect reliably.

Evidence (verbatim from paper)

We use SceneEval*(Tam et al., [2025])* with the following metrics: CNT (object count), ATR (object attributes), OOR (object-object relationships), OAR (object-architecture relationships), ACC (accessibility), NAV (navigability), and OOB (out-of-bounds). We note that these VLM-based metrics have limitations including false positives and negatives (Appendix[L.2]). We add two physics metrics using Drake*([Tedrake et al., 2019])* to evaluate simulation-readiness: COL (collision rate) and STB (static equilibrium).

Citation

@misc{pfaff2026scenexmith,
  title={SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes},
  author={Pfaff et al. (2026)},
  year={2026},
  note={arXiv:2602.09153}
}

• arXiv: 2602.09153