Pharos Benchmark Eval

pharos-benchmark-eval

On the Limits of Tabular Hardness Metrics for Deep RL: A Study with the Pharos Benchmark — Conserva et al. (2025) (arXiv:2509.17092, 2025)

What this evaluates

Evaluates whether traditional tabular reinforcement learning hardness metrics (MDP diameter, suboptimality gaps, effective horizon) can predict the sample efficiency and performance of deep RL agents across different observation modalities and environment scales.

Datasets

• Pharos Benchmark — total ?; splits: instances (-1)

Metrics

• cumulative regret (primary) — range: other
  • Sum of the difference between the optimal reward and the agent's reward at each step over the training horizon. Lower values indicate better performance.
• $R^{2}$ — range: [0, 1]
  • Coefficient of determination measuring the proportion of variance in cumulative regret explained by tabular hardness metrics via linear regression.

Input / output format

Input: Normalized state vectors or raw pixel images representing the environment state.

Output: Discrete action selected by the DQN policy at each timestep.

Scoring recipe

# 1. Train DQN for 50k (small) or 600k (large) steps, average over 5 seeds
# 2. Compute cumulative regret per instance
regret = sum(optimal_reward - agent_reward_t for t in range(steps))
# 3. Fit linear model: Regret ~ intercept + rep_type + env_class + log(eff_horizon) + log(sub_gaps) + log(diameter)
model = LinearRegression().fit(X_tabular_metrics, regret)
# 4. Evaluate fit
r2 = model.score(X_tabular_metrics, regret)

Common pitfalls

• Assuming tabular hardness metrics (diameter, suboptimality gaps) generalize to pixel-based observations without accounting for representation learning difficulty.
• Using a single global linear model across all environments and modalities instead of splitting by representation type or environment class.
• Ignoring the impact of observation modality (image vs. vector) as a dominant source of task difficulty.

Evidence (verbatim from paper)

The $R^{2}$ score of this model is $0.09$, and the fitted vs actual plot is shown in Figure [4]. The poor fit of this model and the absence of any statistical significance in the model coefficient indicate that tabular hardness measures are not able to capture the hardness of the non-tabular task in a way that generalizes across environment classes and representation types.

Citation

@misc{conserva2025pharos,
  title={On the Limits of Tabular Hardness Metrics for Deep RL: A Study with the Pharos Benchmark},
  author={Conserva et al. (2025)},
  year={2025},
  note={arXiv:2509.17092}
}

• arXiv: 2509.17092