Bark Multi Agent Behavior Eval

bark-multi-agent-behavior-eval

BARK: Open Behavior Benchmarking in Multi-Agent Environments — Bernhard et al. (2020) (arXiv:2003.02604, 2020)

What this evaluates

Evaluates the robustness of autonomous driving behavior planners (MCTS, RL, IDM, MOBIL) in interactive multi-agent traffic. It probes how well models handle prediction inaccuracies, parameter variations, and complex merging constraints without fine-tuning.

Datasets

• BARK Sampling Scenarios — total 2400; splits: test (2400); repo https://github.com/bark-simulator/bark
• INTERACTION — total 1; splits: test (1)

Metrics

• collision_rate (primary) — range: percent
  • Percentage of scenarios resulting in a collision out of the total evaluated scenarios. Calculated as (number of collisions / total scenarios) * 100.
• success_rate — range: percent
  • Percentage of scenarios where the controlled agent successfully reaches the goal within the maximum allowed simulation steps.
• average_steps — range: other
  • Mean number of simulation steps taken per scenario, computed only over successfully completed runs.

Input / output format

Input: Initial traffic state including ego vehicle position/velocity, surrounding vehicles' positions/velocities, map geometry, and goal definition (e.g., target lane). For simulation benchmarks, the full recorded scenario sequence with other agents' trajectories.

Output: Action sequence (steering, acceleration, braking) or lane-change decision per time step. Evaluation outputs binary outcomes (collision, success, exceeded_time) or final trajectory.

Scoring recipe

def score(predictions, gold):
    collisions = successes = exceeded = 0
    total_steps = 0
    for pred in predictions:
        if pred.collided: collisions += 1
        elif pred.reached_goal:
            successes += 1
            total_steps += pred.steps
        else: exceeded += 1
    return {
        'collision_rate': collisions / len(predictions),
        'success_rate': successes / len(predictions),
        'avg_steps': total_steps / successes if successes > 0 else 0
    }

Common pitfalls

• Collision metrics alone do not capture scenario deviation; a model might avoid collisions but drastically alter traffic flow or violate ordering constraints.
• Parameter variations (e.g., IDM time headway) are applied to surrounding agents to test prediction robustness, not the ego planner's direct parameters.
• Results are highly sensitive to the fixed maximum simulation steps (30) and time step resolution (0.1s–0.2s), which truncate long-horizon behaviors.

Evidence (verbatim from paper)

Figure 5 gives the percentages of each scenario run with the controlled agent reaching the goal, colliding, or exceeding the maximum allowed simulation time steps (>30), as well as the average number of simulation steps in the event that the goal is reached. It is clear that the collision rate rises as the prediction error increases.

Citation

@misc{bernhard2020bark,
  title={BARK: Open Behavior Benchmarking in Multi-Agent Environments},
  author={Bernhard et al. (2020)},
  year={2020},
  note={arXiv:2003.02604}
}

• arXiv: 2003.02604