Fogmachine Eval

fogmachine-eval

FOGMACHINE -- Leveraging Discrete-Event Simulation and Scene Graphs for Modeling Hierarchical, Interconnected Environments under Partial Observations from Mobile Agents — Ohnemus et al. (2025) (arXiv:2510.09483, 2025)

What this evaluates

Evaluates a discrete-event simulation framework that fuses dynamic scene graphs with urban environments to model hierarchical, interconnected spaces under partial observability. It probes the simulator's capacity to reproduce emergent temporal behaviors, the accuracy of state reconstruction when agent views are sparse, and the computational efficiency of the underlying simulation engine.

Datasets

• FOGMACHINE Scenarios (Bruchsal, Wenningstedt, Trier) — total ?; splits: (unstated)

Metrics

• RTF (primary) — range: other
  • Real-Time Factor, defined as the ratio of simulated time to wall-clock time. Higher values indicate faster simulation execution.
• Up-to-date node share — range: percent
  • The percentage of simulation time during which all objects at path network nodes are correctly reflected in the observed scene graph.
• Average Task Delay — range: percent
  • Normalized difference between predicted and actual task completion times, averaged over all executed tasks. Formula: d = (t_pred - t_true) / t_true.
• Inter-observation time — range: other
  • The average interval between two consecutive observations of the same node, measuring observation frequency.

Input / output format

Input: Dynamic scene graph G, observed scene graph G_obs[t], static path network S, and task parameters including true completion times.

Output: Predicted task completion times t_pred and planner trajectories generated using either the static path network S or the observed dynamic scene graph G_obs.

Scoring recipe

RTF = simulated_time / wall_clock_time
up_to_date_time = sum(1 for t in timeline if all_objects_correct_in_obs(t))
up_to_date_share = (up_to_date_time / total_sim_time) * 100
task_delays = []
for task in executed_tasks:
    d = (task.t_pred - task.t_true) / task.t_true
    task_delays.append(d)
avg_task_delay = (sum(task_delays) / len(task_delays)) * 100
inter_obs_times = [t2 - t1 for t1, t2 in consecutive_obs_per_node]
avg_inter_obs_time = sum(inter_obs_times) / len(inter_obs_times)

Common pitfalls

• The first 48 hours of each 22-day simulation are omitted as a burn-in period; including them would distort emergent behavior and observability metrics.
• True baseline distributions are sampled over 40,000 days, not the 22-day evaluation runs, so comparing short-run metrics directly to true distributions without accounting for this sampling difference is invalid.
• Metrics are reported as mean ± standard deviation across 5 replications per scenario; reporting single-run values or ignoring the standard deviation violates the evaluation protocol.

Evidence (verbatim from paper)

To capture the impact of partial observability, we quantify the mismatch between the true state G and the observed state G_obs[t]. We employ several observability metrics: the inter-observation time, i.e., the interval between two consecutive observations of the same node, with the average inter-observation time per node providing a measure of observation frequency; the up-to-date node share, i.e., the percentage of simulation time during which all objects at nodes v_P^(i) in V_P are correctly reflected in G_obs[t] ; and the task delay, defined as the normalized difference between predicted and actual task completion times, d = (t_pred - t_true) / t_true. Averaging d over all executed tasks yields the average task delay.

Citation

@misc{ohnemus2025fogmachine,
  title={FOGMACHINE -- Leveraging Discrete-Event Simulation and Scene Graphs for Modeling Hierarchical, Interconnected Environments under Partial Observations from Mobile Agents},
  author={Ohnemus et al. (2025)},
  year={2025},
  note={arXiv:2510.09483}
}

• arXiv: 2510.09483