Super Ddqn Eval

super-ddqn-eval

Selectively Sharing Experiences Improves Multi-Agent Reinforcement Learning — Gerstgrasser et al. (2023) (arXiv:2311.00865, 2023)

What this evaluates

Evaluates a decentralized multi-agent reinforcement learning algorithm that selectively shares high-temporal-difference-error experiences. It probes cooperative and competitive multi-agent coordination, credit assignment, and communication efficiency in anonymous environments with separate per-agent reward signals.

Datasets

• PettingZoo (Pursuit, Battle, Adversarial-Pursuit) — total ?; splits: train (-1); repo https://github.com/Farama-Foundation/PettingZoo

Metrics

• total mean episode reward (primary) — range: other
  • Average of the sum of rewards obtained by all agents (or the designated sharing team) across all evaluation episodes.

Input / output format

Input: Per timestep: individual observations for each agent, joint action space, and per-agent reward signals. Environment state evolves according to multi-agent dynamics.

Output: Action selected by each agent's independent policy network at each timestep.

Scoring recipe

def compute_metric(rollouts):
    episode_rewards = []
    for rollout in rollouts:
        episode_rewards.append(sum(rollout.rewards))
    return sum(episode_rewards) / len(episode_rewards)

Common pitfalls

• Pre-training opposing teams in adversarial domains (Battle, Adversarial-Pursuit) before main training is strictly required; failing to do so breaks the evaluation setup.
• Sharing all experiences indiscriminately ('all' bandwidth) actually degrades performance compared to no sharing, contrary to intuition.
• Parameter sharing is a centralized baseline that shares policy weights, not joint control; confusing the two misrepresents the comparison.

Evidence (verbatim from paper)

For Pursuit, performance is the total mean episode reward from all agents. For Battle and Adversarial-Pursuit, performance is the total mean episode reward from all agents in the sharing team (blue team in Battle, prey team in Adversarial-Pursuit).

Citation

@misc{gerstgrasser2023super,
  title={Selectively Sharing Experiences Improves Multi-Agent Reinforcement Learning},
  author={Gerstgrasser et al. (2023)},
  year={2023},
  note={arXiv:2311.00865}
}

• arXiv: 2311.00865