Swarmbench Eval

swarmbench-eval

Benchmarking LLMs' Swarm intelligence — Ruan et al. (2025) (arXiv:2505.04364, 2025)

What this evaluates

This benchmark evaluates emergent decentralized coordination in LLM-driven multi-agent systems under strict local perception and communication constraints. It simulates five canonical swarm tasks—Pursuit, Synchronization, Foraging, Flocking, and Transport—in a 2D grid environment to probe whether LLMs can form adaptive group strategies and execute robust long-range planning without global information.

Datasets

• SwarmBench — total ?; splits: test (-1); repo https://github.com/x66ccff/swarmbench

Metrics

• Performance (primary) — range: other
  • Average task performance score across five independent simulation runs per model and task. Task-specific success conditions vary by environment (e.g., target capture rate, synchronization accuracy, food collection, flock cohesion, object transport).

Input / output format

Input: A 5x5 local grid view (k×k observation window) and local communication messages from neighboring agents in the preceding round, provided within a 2D grid environment.

Output: Discrete action selection per agent per time step (e.g., movement direction, interaction/push, or communication broadcast) based on local observations and received messages.

Scoring recipe

def compute_performance(actions, task_type, n_runs=5):
    scores = []
    for _ in range(n_runs):
        env = SwarmBenchEnvironment(task_type, view_size=5)
        for step in range(max_steps):
            for agent in agents:
                obs = env.get_local_view(agent)
                msgs = env.get_received_messages(agent)
                action = model.predict(obs, msgs)
                env.step(action)
            scores.append(env.get_task_score(task_type))
    return mean(scores)

Common pitfalls

• Assuming explicit message content directly drives task success; the paper shows physical dynamics and implicit coordination are stronger predictors.
• Interpreting high permutation importance of messages as strategic group-level success; messages influence local tactical decisions but do not guarantee optimal global outcomes.
• Assuming larger perception ranges (e.g., k=7) always improve performance; the paper notes diminishing returns or degradation due to increased prompt complexity and noise.

Evidence (verbatim from paper)

Performance, averaged over five runs per model/task, reveals significant variation across both LLMs and tasks, highlighting the inherent difficulty of achieving decentralized coordination under strict local constraints.

Citation

@misc{ruan2025swarmbench,
  title={Benchmarking LLMs' Swarm intelligence},
  author={Ruan et al. (2025)},
  year={2025},
  note={arXiv:2505.04364}
}

• arXiv: 2505.04364