Dynamic House Simulator Eval

dynamic-house-simulator-eval

Modeling Dynamic Environments with Scene Graph Memory — Kurenkov et al. (2023) (arXiv:2305.17537, 2023)

What this evaluates

Evaluates an agent's ability to perform temporal link prediction and object search in partially observable, dynamic environments by predicting object locations, ranking location likelihoods, and navigating to objects sequentially.

Datasets

• Dynamic House Simulator — total ?; splits: (unstated)

Metrics

• accuracy — range: [0, 1]
  • Binary metric indicating whether the predicted furniture node is correctly connected to the target object in the ground-truth scene graph.
• NDCG (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain, a ranking metric that measures the quality of predicted location likelihoods by discounting gains based on position in the ranked list.
• steps — range: other
  • Count of sequential location choices made by the agent until it correctly identifies the target node.

Input / output format

Input: Partial, time-stamped scene graph memory (SGM) containing observed nodes and edges, with specific query nodes/edges annotated for prediction.

Output: Task 1: Single predicted furniture node. Task 2: Ranked list of location likelihoods for each queried object. Task 3: Sequence of location choices until success.

Scoring recipe

def score(predictions, gold, task):
    if task == 'predict_location':
        return 1.0 if pred_node in gold.connected_nodes else 0.0
    elif task == 'predict_likelihood':
        return ndcg_score(gold.ranks, pred_ranks)
    elif task == 'find_object':
        return len(pred_sequence)
    return 0.0

Common pitfalls

• The 'Find Object' task uses a static environment during the search phase, meaning the metric only measures search efficiency rather than dynamic reasoning.
• NDCG evaluates ranking quality over exact probability calibration, so models are not penalized for poorly scaled likelihood values.
• Training data distribution depends on the heuristic policies used during data collection, which may bias the model toward specific exploration behaviors.

Evidence (verbatim from paper)

We define three tasks with associated metrics as the basis of our experiments. Predict Object Location: At every timestep, the agent must predict the location (furniture node) of an object with a particular description. The agent is then able to observe the node it has predicted and its associated object nodes, regardless of whether its output is correct. The metric for the task is accuracy – whether the agent correctly predicted a furniture node that is connected to an object with the correct description. This task is designed to evaluate how well an embodied agent may work in practice in an unknown environment; predicting the correct location of a given query object is essential for downstream tasks like household chores. Predict Relative Location Likelihood: At every timestep, the agent is queried for multiple objects and is required to predict the likelihood of each location that each object can be at. The metric for the task is the Normalized Discounted Cumulative Gain (NDCG), a popular method for measuring the quality of a set of search results. We choose this metric as we primarily care about the agent correctly predicting the ranking of the location options rather than the

Citation

@misc{kurenkov2023modeling,
  title={Modeling Dynamic Environments with Scene Graph Memory},
  author={Kurenkov et al. (2023)},
  year={2023},
  note={arXiv:2305.17537}
}

• arXiv: 2305.17537