AAMAS Reproducibility
Use this before submission and again before camera-ready. The reproducibility question at AAMAS
is not only "can I rerun the model" but "can I reproduce the interaction - the same agents,
the same game, the same emergent outcome."
Evidence map
- Map each theorem, mechanism property, convergence claim, and empirical interaction claim to a
verifiable location in the paper, appendix, supplement, or artifact.
- For theory, state the game, the information structure, the solution concept, assumptions,
proof dependencies, and failure modes clearly enough for a game theorist.
- For experiments, report the environment, number of agents, opponent/population set, training
regime, evaluation opponents, metrics, hyperparameter ranges, chosen settings, seeds,
repeated runs, compute, and runtime.
- For small or noisy strategic differences, add uncertainty: standard errors, confidence
intervals, or paired tests over seeds and over opponents.
- Explain any missing code or environment honestly, and describe how a reader could reproduce
the interaction in principle.
- Keep the artifact consistent with the paper; a claim the artifact cannot demonstrate is a
review-risk multiplier.
Claim-to-evidence audit table
| Claim |
Pure-theory answer |
Learning-plus-game answer |
| Solution concept reached |
Proof with the game and information structure stated |
Plus convergence curves under other agents' adaptation |
| Opponents / population |
NA if fully analytical |
The exact opponent set and how it was chosen |
| Seeds and variance |
NA for deterministic results |
Required for every stochastic curve and payoff table |
| Compute |
NA |
Hardware, per-run time, and total number of self-play runs |
Claiming an equilibrium result while the evaluation only shows two fixed agents playing once is
the recognizable AAMAS gap: reviewers read the mismatch between the strategic claim and the
thinness of the interaction evidence as carelessness about the rest.
Vignette: a MARL-plus-convergence paper
Consider a submission proving convergence to a coarse-correlated equilibrium in a repeated game,
validated by self-play. Its reproducibility spine: the game generator and payoff scale, the
learning rule and its step sizes, the opponent set, the replication seeds, the convergence
metric, and one honest sentence on the regime where convergence is only empirical, not proved.
Degrees of reproducibility
- Turnkey: one command reruns the game and regenerates each convergence figure from logged
seeds.
- Scripted: scripts exist but require documented manual steps or an external environment.
- Descriptive: prose detailed enough that a competent reader could rebuild the game and agents.
For AAMAS, the strategic core should be turnkey because reviewers actually re-run small games;
large real-world or human-in-the-loop pipelines may stay scripted with deviations documented.
Stating the achieved level honestly beats overpromising turnkey behavior that fails on a clean
machine.
Output format
[Claim inventory] <claim -> evidence location>
[Artifact consistency] complete / inconsistent / missing
[Interaction reproducibility gaps] <game/opponents/seeds/uncertainty/compute>
[Paper fixes] <must appear in main PDF>
[Supplement fixes] <appendix or artifact additions>
Source: brycewang-stanford/Awesome-Journal-Skills → AAMAS-Skills/skills/aamas-reproducibility/SKILL.md
1---2name: aamas-reproducibility3description: Use when strengthening AAMAS reproducibility evidence for interaction claims, including proofs and game definitions, opponent and population sets, self-play protocols, random seeds, compute, uncertainty on strategic outcomes, baselines, and consistency between what the paper claims about agents and what the artifact can actually show.4---567# AAMAS Reproducibility89Use this before submission and again before camera-ready. The reproducibility question at AAMAS10is not only "can I rerun the model" but "can I reproduce the *interaction* - the same agents,11the same game, the same emergent outcome."1213## Evidence map1415- Map each theorem, mechanism property, convergence claim, and empirical interaction claim to a16 verifiable location in the paper, appendix, supplement, or artifact.17- For theory, state the game, the information structure, the solution concept, assumptions,18 proof dependencies, and failure modes clearly enough for a game theorist.19- For experiments, report the environment, number of agents, opponent/population set, training20 regime, evaluation opponents, metrics, hyperparameter ranges, chosen settings, seeds,21 repeated runs, compute, and runtime.22- For small or noisy strategic differences, add uncertainty: standard errors, confidence23 intervals, or paired tests over seeds and over opponents.24- Explain any missing code or environment honestly, and describe how a reader could reproduce25 the interaction in principle.26- Keep the artifact consistent with the paper; a claim the artifact cannot demonstrate is a27 review-risk multiplier.2829## Claim-to-evidence audit table3031| Claim | Pure-theory answer | Learning-plus-game answer |32|---|---|---|33| Solution concept reached | Proof with the game and information structure stated | Plus convergence curves under other agents' adaptation |34| Opponents / population | NA if fully analytical | The exact opponent set and how it was chosen |35| Seeds and variance | NA for deterministic results | Required for every stochastic curve and payoff table |36| Compute | NA | Hardware, per-run time, and total number of self-play runs |3738Claiming an equilibrium result while the evaluation only shows two fixed agents playing once is39the recognizable AAMAS gap: reviewers read the mismatch between the strategic claim and the40thinness of the interaction evidence as carelessness about the rest.4142## Vignette: a MARL-plus-convergence paper4344Consider a submission proving convergence to a coarse-correlated equilibrium in a repeated game,45validated by self-play. Its reproducibility spine: the game generator and payoff scale, the46learning rule and its step sizes, the opponent set, the replication seeds, the convergence47metric, and one honest sentence on the regime where convergence is only empirical, not proved.4849## Degrees of reproducibility5051- Turnkey: one command reruns the game and regenerates each convergence figure from logged52 seeds.53- Scripted: scripts exist but require documented manual steps or an external environment.54- Descriptive: prose detailed enough that a competent reader could rebuild the game and agents.5556For AAMAS, the strategic core should be turnkey because reviewers actually re-run small games;57large real-world or human-in-the-loop pipelines may stay scripted with deviations documented.58Stating the achieved level honestly beats overpromising turnkey behavior that fails on a clean59machine.6061## Output format6263```text64[Claim inventory] <claim -> evidence location>65[Artifact consistency] complete / inconsistent / missing66[Interaction reproducibility gaps] <game/opponents/seeds/uncertainty/compute>67[Paper fixes] <must appear in main PDF>68[Supplement fixes] <appendix or artifact additions>69```7071---7273**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `AAMAS-Skills/skills/aamas-reproducibility/SKILL.md`
thedixitjain