Scenicrules Eval

scenicrules-eval

ScenicRules: An Autonomous Driving Benchmark with Multi-Objective Specifications and Abstract Scenarios — Chang et al. (2026) (arXiv:2602.16073, 2026)

What this evaluates

Evaluates autonomous driving agents on their ability to navigate stochastic traffic scenarios while satisfying a hierarchical set of multi-objective specifications. It probes how well agents balance conflicting goals like collision avoidance, road compliance, passenger comfort, and progress under varying priority constraints.

Datasets

• ScenicRules Benchmark — total ?; splits: test (-1); repo https://github.com/BerkeleyLearnVerify/ScenicRules

Metrics

• Violation Score (VS) (primary) — range: other
  • Quantifies the extent of violation for each of the 19 formalized rules. Computed using Signal Temporal Logic (STL) robustness metrics or custom objective functions over the trajectory. A positive value indicates a violation, with larger values denoting greater severity.

Input / output format

Input: Simulation trajectory data including ego and agent positions (polygons), velocities, accelerations, and environmental maps (drivable areas, lane boundaries, target regions) over the simulation time interval [T1, T2].

Output: A violation score (VS) value for each evaluated rule, or a dictionary mapping rule IDs to their respective VS values.

Scoring recipe

def compute_vs(traj, env, rule_id):
    # Extract trajectory states and environment parameters
    p_ego, v_ego, a_ego = traj.positions, traj.velocities, traj.accelerations
    vs = 0.0
    if rule_id == 3:  # Staying within drivable area
        out_area = max_t(area(p_ego(t) \ env.drivable_area))
        dist_term = max_t(dist(p_ego(t), env.drivable_area))
        vs = out_area + dist_term**2
    elif rule_id == 13:  # Speed limit
        vs = max(max_t(norm(v_ego(t)) - env.speed_limit), 0)**2
    elif rule_id == 15:  # Lane centering
        vs = sum_t(dist(centroid(p_ego(t)), env.centerline(t)))
    # ... other rules follow similar pattern using STL robustness or objective functions
    return vs

Common pitfalls

• Using only STL robustness without distance-based terms may fail to differentiate violation severity once an agent is fully outside a boundary.
• Default priority groupings are reference baselines; actual evaluation requires context-specific priority permutations (e.g., emergency vs. dense traffic).
• Choosing between max-vs-sum violation aggregation changes which trajectories are considered optimal.

Evidence (verbatim from paper)

Each rule is defined by a logical formula together with a corresponding violation score (VS) that measures the extent of violation. A positive VS indicates a violation, and larger values correspond to more severe violations.

Citation

@misc{chang2026scenicrules,
  title={ScenicRules: An Autonomous Driving Benchmark with Multi-Objective Specifications and Abstract Scenarios},
  author={Chang et al. (2026)},
  year={2026},
  note={arXiv:2602.16073}
}

• arXiv: 2602.16073