Flatland 2020 Eval

flatland-2020-eval

Flatland Competition 2020: MAPF and MARL for Efficient Train Coordination on a Grid World — Laurent et al. (2021) (arXiv:2103.16511, 2021)

What this evaluates

Evaluates multi-agent coordination and path planning for train rescheduling in a dynamic grid-world railway simulation. It probes the ability of agents to adapt to partial observability, handle congestion, and coordinate under switching constraints and dynamic disruptions.

Datasets

• Flatland Competition 2020 — total ?; splits: test (-1)

Metrics

• overall score (primary) — range: other
  • A composite metric that balances two competing goals: maximizing the percentage of trains that reach their targets (completion rate) and minimizing the total time required to bring them there. Exact weighting is not specified in the text.

Input / output format

Input: Grid-based railway simulation environment with dynamic disruptions; agents receive observations such as local tree structures, global conflict graphs, or hand-crafted features depending on the solution.

Output: Action commands to coordinate train movements and reschedule paths in real-time to avoid collisions and reach targets.

Scoring recipe

def compute_score(completed_trains, total_trains, total_time):
    completion_rate = completed_trains / total_trains
    # Score balances completion rate and time minimization
    # Higher completion and lower time yield a higher overall score
    return composite_metric(completion_rate, total_time)

Common pitfalls

• The final ranking depends on a composite score, not just completion rate; a team with fewer completed environments can rank higher if their average score per environment is better.
• Both centralized and decentralized execution paradigms achieved comparable performance, so algorithm choice (centralized vs decentralized) is not a strict performance predictor.
• The environment features dynamic disruptions and switching constraints, requiring real-time adaptation rather than static path planning.

Evidence (verbatim from paper)

Table 1 shows their overall scores, completion percentages and number of encountered environments. The OR solution (An_Old_Driver) tackled the most environments with a total of 363, followed by the winning RL solution (JBR_HSE) with 336. Interestingly, the second RL solution (Netcetera) tackled 1 environment less (229) than the fourth RL solution (MARMot-Lab-NUS, 230), but scored better on average resulting in a better ranking. This perfectly illustrates the two competing goals of the challenge: maximizing the number of agents that reach their targets vs minimising the time needed to bring the agents there.

Citation

@misc{laurent2021flatland,
  title={Flatland Competition 2020: MAPF and MARL for Efficient Train Coordination on a Grid World},
  author={Laurent et al. (2021)},
  year={2021},
  note={arXiv:2103.16511}
}

• arXiv: 2103.16511