Model Recovery Eval

model-recovery-eval

Model Recovery at the Edge under Resource Constraints for Physical AI — Xu et al. (2025) (arXiv:2512.02283, 2025)

What this evaluates

This benchmark evaluates the accuracy and hardware efficiency of neural flow-based architectures for recovering underlying dynamics from time-series data. It probes the model's ability to estimate parameters of nonlinear dynamical systems while measuring computational resource constraints like runtime, power, and memory footprint on edge hardware.

Datasets

• Chaotic Lorenz — total ?; splits: test (-1)
• F8 Cruiser — total ?; splits: test (-1)
• Lotka Volterra — total ?; splits: test (-1)
• Pathogenic Attack System — total ?; splits: test (-1)
• Automated Insulin Delivery (OhioT1D) — total 200; splits: test (200)

Metrics

• reconstruction MSE (primary) — range: [0, inf)
  • Mean Square Error between the estimated parameters and the ground truth values.
• runtime — range: seconds
  • Execution time measured in seconds using the time library.
• energy — range: Joules
  • Total energy consumption in Joules, calculated as average power multiplied by runtime.
• DRAM footprint — range: MB
  • Memory usage in megabytes measured using the psutil library.

Input / output format

Input: Time-series sequences of system states (e.g., glucose levels, population counts, chaotic variables) sampled at fixed intervals.

Output: Estimated parameters defining the underlying nonlinear dynamical equations, plus hardware metrics (runtime, power, DRAM, energy) for the deployed accelerator.

Scoring recipe

def compute_mse(est_params, true_params):
    return np.mean((est_params - true_params) ** 2)

def compute_energy(avg_power_w, runtime_s):
    return avg_power_w * runtime_s

Common pitfalls

• Confusing parameter estimation error (reconstruction MSE) with state-trajectory prediction error.
• Assuming energy and memory are always positively correlated; the paper explicitly demonstrates an inverse tradeoff for MR systems.
• Treating hardware metrics (power, DRAM) as algorithm-intrinsic rather than platform-dependent (FPGA vs. GPU).

Evidence (verbatim from paper)

Compared to SOTA MR methods such as EMILY and PINN+SR, Table 3 shows that MERINDA achieves comparable or even lower errors across four benchmark applications. Accuracy is measured using Mean Square Error between the estimated parameters and the ground truth values.

Citation

@misc{xu2025modelrecovery,
  title={Model Recovery at the Edge under Resource Constraints for Physical AI},
  author={Xu et al. (2025)},
  year={2025},
  note={arXiv:2512.02283}
}

• arXiv: 2512.02283