Neural Mmo Eval

neural-mmo-eval

Benchmarking Robustness and Generalization in Multi-Agent Systems: A Case Study on Neural MMO — Chen et al. (2023) (arXiv:2308.15802, 2023)

What this evaluates

This benchmark evaluates the robustness and generalization of multi-agent reinforcement learning policies in a large-scale, open-ended simulation. It probes a model's ability to cooperate with teammates and compete against unknown opponents or fixed baselines across varying difficulty levels and dynamic environments.

Datasets

• Neural MMO — total ?; splits: test (-1)

Metrics

• TrueSkill (primary) — range: other
  • A Bayesian skill rating system that computes a relative strength score for each submission based on match outcomes and achievements across approximately 1000 PvP matches, accounting for opponent strength variability.
• Top1Ratio — range: [0, 1]
  • The ratio of games won (i.e., achieving the highest team score among 16 teams) over 10 PvE matches against built-in AI baselines.
• Achievement — range: other
  • A multi-task point score awarded for completing tasks at three difficulty levels (4 for easy, 10 for normal, 21 for hard), with only the highest tier completed per category counting toward the total.

Input / output format

Input: Per-step observations from the Neural MMO environment (128x128 map state), controlling a team of 8 agents in free-for-all matches against 15 other teams (PvP) or built-in AI (PvE).

Output: Action commands for each of the 8 agents per environment step, executed over 1024 steps per match.

Scoring recipe

# PvE Track
wins = 0
for _ in range(10):
    scores = simulate_match(policy, baseline_ai, teams=16, steps=1024)
    if scores[policy_id] == max(scores):
        wins += 1
top1_ratio = wins / 10

# PvP Track
# Run ~1000 matches against random 16-submission pools
# Compute TrueSkill ratings based on match outcomes/achievements
trueskill_scores = compute_trueskill(match_results)

Common pitfalls

• Using mean achievement score for PvP evaluation instead of TrueSkill, which fails to account for opponent strength variability and misrepresents robustness.
• Assuming all 8 agents on a team must coordinate on a single task; the protocol awards points if any single agent completes the highest tier of a task, encouraging specialization.
• Confusing the PvE and PvP evaluation pipelines: PvE provides immediate feedback against fixed AI baselines, while PvP runs weekly against other submissions.

Evidence (verbatim from paper)

For the PvP track evaluation procedure, we randomly select 16 submissions from all qualified submissions to start a PvP match. In the final evaluation, each submission will participate in approximately 1000 matches. The mean achievement score is not a good evaluation metric due to the variability of opponents. For example, model A gets a high score against weaker opponents and a low score against stronger opponents, while model B gets an above-average score against all levels of opponents. In this case, the mean achievement scores of the two models may be close, but it is obvious that model B is more robust. To more accurately measure the relative strength of the models, we use TrueSkill [8] to compute scores for each submission.

Citation

@misc{chen2023neuralmmo,
  title={Benchmarking Robustness and Generalization in Multi-Agent Systems: A Case Study on Neural MMO},
  author={Chen et al. (2023)},
  year={2023},
  note={arXiv:2308.15802}
}

• arXiv: 2308.15802