Findingdory Eval

findingdory-eval

FindingDory: A Benchmark to Evaluate Memory in Embodied Agents — Yadav et al. (2025) (arXiv:2506.15635, 2025)

What this evaluates

This benchmark evaluates long-term memory and spatio-temporal reasoning in embodied agents. It requires agents to recall specific past interactions from a video history to select goal frames and navigate to target entities in dynamic, photorealistic environments over long-horizon tasks.

Datasets

• FindingDory — total 60; splits: test (-1)

Metrics

• LL-SR (primary) — range: [0, 1]
  • Low Level Success Rate. Measures whether the agent successfully finds the right target entities by selecting a valid goal frame from its interaction history. Exact success criteria and thresholds are defined in the paper's appendix.
• LL-SPL — range: [0, 1]
  • Low Level Success-weighted-by-Path-Length. Evaluates if the agent selects and navigates to the optimal entity using the shortest possible path. Computed as success indicator multiplied by the ratio of optimal path length to actual path length.
• HL-SR — range: [0, 1]
  • High-Level Policy Success Rate. Measures the accuracy of the high-level policy in selecting the correct and closest goal frames for navigation from the interaction history.
• HL-SPL — range: [0, 1]
  • High-Level Policy SPL. Measures the efficiency of the high-level policy in selecting the correct and closest goal frames, weighted by path optimality.
• DTG-SR — range: [0, 1]
  • Distance-to-Goal-Only SR. A relaxed variant that ignores whether the target object must be visible in the selected frame, focusing only on navigational distance.
• SC-SR — range: [0, 1]
  • Semantic Coverage SR. A relaxed variant that ignores spatial proximity, measuring only semantic correctness of the selected goal.

Input / output format

Input: Full interaction video (frames annotated with index and time of day) combined with a natural language task instruction. For the textual memory baseline, video frames are chunked and summarized into text before processing.

Output: A single integer representing the predicted frame index from the interaction history that corresponds to a viable goal state. The low-level policy then outputs discrete actions: MOVE-FORWARD, TURN-RIGHT, TURN-LEFT, or STOP.

Scoring recipe

def evaluate(pred_frame_idx, valid_frames, goal_entity, actual_path_len, optimal_path_len):
    # Check if predicted frame is in the list of acceptable target frames
    is_valid = pred_frame_idx in valid_frames
    # Check if navigation to goal entity succeeds from that frame
    success = is_valid and reaches_target(pred_frame_idx, goal_entity)
    
    # LL-SR
    ll_sr = 1.0 if success else 0.0
    
    # LL-SPL
    if success:
        ll_spl = min(1.0, optimal_path_len / actual_path_len)
    else:
        ll_spl = 0.0
        
    return ll_sr, ll_spl

Common pitfalls

• The high-level action space is constrained to frames observed during experience collection, making some tasks theoretically unsolvable without an oracle or teleportation policy.
• Multiple consecutive frames may validly represent a goal state; evaluators must check predictions against a full list of acceptable frames rather than a single ground-truth index.
• Agents may rely on heuristic shortcuts instead of true memory recall; the benchmark design explicitly requires recalling specific past events in dynamic scenes to prevent this.

Evidence (verbatim from paper)

We benchmark agent performance on the FindingDory tasks using Low Level Success Rate (LL-SR), assessing whether the agent finds the right target entities, and Low Level Success-weighted-by-Path-Length (LL-SPL), evaluating if the agent selects and navigates to the optimal entity using the shortest possible path. Since we use a hierarchical baseline, we additionally introduce a High-Level Policy Success Rate (HL-SR), and High-Level Policy SPL (HL-SPL) which measures the accuracy and efficiency of the high-level policy in selecting the correct and closest goal frames for navigation.

Citation

@misc{yadav2025findingdory,
  title={FindingDory: A Benchmark to Evaluate Memory in Embodied Agents},
  author={Yadav et al. (2025)},
  year={2025},
  note={arXiv:2506.15635}
}

• arXiv: 2506.15635