Smarts Eval

smarts-eval

On Multi-Agent Deep Deterministic Policy Gradients and their Explainability for SMARTS Environment — Mittal et al. (2023) (arXiv:2301.09420, 2023)

What this evaluates

Evaluates multi-agent reinforcement learning algorithms in a simulated urban driving environment, measuring scenario completion, episode duration, human-like driving fidelity, and traffic rule compliance.

Datasets

• SMARTS (NeurIPS Competition Track-1) — total ?; splits: test (-1)

Metrics

• Completion (primary) — range: [0, 1]
  • Success or completion rate per episode. The text ambiguously states 'total number of crashes', but table values (0.24–0.72) indicate a normalized success rate.
• Time — range: steps
  • Total number of simulation steps taken by each agent (red cars) per episode.
• Humanness — range: other
  • Average of total distance to obstacles, angular jerk, linear jerk, and lane center offset per episode.
• Rules — range: other
  • Total number of traffic rules violated per episode (e.g., illegal lane changes, wrong way, speeding).

Input / output format

Input: Per-episode simulator state observations and agent positions in the SMARTS urban driving environment.

Output: Per-step control actions (speed, direction, waypoints) for each agent per episode.

Scoring recipe

def compute_metrics(episode_actions, episode_states):
    completion = 1.0 if episode_actions.success else 0.0
    time_steps = len(episode_actions)
    humanness = np.mean([
        distance_to_obstacles(episode_states),
        angular_jerk(episode_actions),
        linear_jerk(episode_actions),
        lane_center_offset(episode_states)
    ])
    rules_violated = count_violations(episode_actions, ['lane_change', 'wrong_way', 'speed_overlimit'])
    return completion, time_steps, humanness, rules_violated

Common pitfalls

• Metrics are averaged across multiple agents within a single episode, leading to higher absolute values than single-agent baselines.
• The textual definition of Completion ('total number of crashes') contradicts the table values (0.24–0.72), which represent a success rate.
• Evaluation is strictly limited to online learning (Task 1) and does not cover offline or multi-task settings.

Evidence (verbatim from paper)

1. Completion This represents the completion of the total scenarios and hence can be calculated as the total number of crashes for each episode 2. Time This metric discusses the total number of steps used by each agent (red cars) for each episode 3. Humannness It refers to how close are our agents to imitating human-level driving scenarios. It is the average of the total distance to obstacles, angular jerk, linear jerk, and lane center offset for each episode 4. Rules It refers to the total number of rules violated such as lane changing, Wrong Way, Speed Overlimit for each episode.

Citation

@misc{mittal2023multiagent,
  title={On Multi-Agent Deep Deterministic Policy Gradients and their Explainability for SMARTS Environment},
  author={Mittal et al. (2023)},
  year={2023},
  note={arXiv:2301.09420}
}

• arXiv: 2301.09420