Neorl2 Eval

neorl2-eval

NeoRL-2: Near Real-World Benchmarks for Offline Reinforcement Learning with Extended Realistic Scenarios — Songyi Gao et al. (arXiv:2503.19267, 2025)

What this evaluates

This benchmark evaluates offline reinforcement learning algorithms on seven near real-world environments featuring time delays, external disturbances, safety constraints, and conservative data collection. It probes whether state-of-the-art offline RL methods can improve upon sub-optimal behavior policies without online exploration, highlighting their robustness to realistic dynamics and safety limits.

Datasets

• NeoRL-2 — total ?; splits: offline_data (-1); repo https://github.com/polixir/NeoRL2

Metrics

• normalized score (0-100) (primary) — range: [0, 100]
  • Computed by normalizing the average return of the evaluated policy against the random policy (minimum bound) and the expert SAC policy (maximum bound) for each environment. Formula: ((avg_return - random_return) / (expert_return - random_return)) * 100.

Input / output format

Input: Offline trajectory datasets (states, actions, rewards) collected from sub-optimal policies (e.g., intermediate SAC checkpoints or PID controllers).

Output: A policy network that maps environment states to actions, evaluated via online rollouts in the target environment.

Scoring recipe

1. Collect baseline returns: random_return = run_policy(RandomPolicy(), env)
2. Collect expert return: expert_return = run_policy(ExpertSACPolicy(), env)
3. Train offline RL policy on dataset
4. Evaluate trained policy over 3 seeds: avg_return = mean([run_policy(trained_policy, env, seed=s) for s in seeds])
5. Normalize score: normalized_score = ((avg_return - random_return) / (expert_return - random_return)) * 100
6. Report mean and standard error across seeds.

Common pitfalls

• Expert policy is explicitly noted as a good policy but not necessarily optimal, so the 100-point bound is an approximation.
• Model-based methods (e.g., MOPO, RAMBO) exhibit high variance across random seeds, making single-run evaluations misleading.
• Evaluation is strictly online (running the trained policy in the environment), but training must remain strictly offline using only the provided dataset.

Evidence (verbatim from paper)

In normalization of data scores (0-100), the scores given by expert policies are used as the maximum bound. ... We run each configuration of the hyperparameters with 3 random seeds and choose the policy at the final training stage to conduct the online test. We report the results from the best hyperparameter across 3 seeds and the 3 seeds are the same for all the algorithms.

Citation

@misc{gao2025neorl2,
  title={NeoRL-2: Near Real-World Benchmarks for Offline Reinforcement Learning with Extended Realistic Scenarios},
  author={Songyi Gao et al.},
  year={2025},
  note={arXiv:2503.19267}
}

• arXiv: 2503.19267