AAAI Experiments
Use this before submission to ensure empirical evidence supports the AI contribution. AAAI
reviewers may come from adjacent AI subfields, so experiments must be interpretable beyond one
benchmark community.
Experiment audit
- Map every experimental block to a claim in the introduction.
- Compare against strong, recent, and fairly tuned baselines.
- Include ablations that isolate mechanisms rather than removing multiple components at once.
- Report uncertainty, variance, and statistical tests when small differences matter.
- Test robustness to data split, prompt, seed, environment, user population, or distribution shift
when relevant.
- For human evaluation, document task, instructions, annotator pool, quality control, aggregation,
and ethics/IRB status.
- Report compute, hardware, data access, model size, and training/inference cost.
Claim-to-evidence ledger
Build this table before adding new experiments. It keeps the AAAI evidence package aligned with
the main text and with the reproducibility checklist.
| Manuscript claim |
Required evidence |
Phase-1 risk if missing |
Checklist hook |
| New AI capability |
benchmark + qualitative failure cases |
broad reviewer sees only engineering |
datasets, metrics, baselines |
| Better mechanism |
single-factor ablations |
gain looks like tuning luck |
ablation and hyperparameter answers |
| Robust deployment |
shift / seed / subgroup stress test |
result seems brittle |
variance, compute, environment |
| Social-impact or safety claim |
stakeholder, harm, and misuse analysis |
ethical claim looks asserted |
ethics, limitations, data access |
For each row, mark ready / weak / missing and name the fastest fix that can be run before the
supplementary-material deadline. Do not leave a claim in the abstract if its evidence row is weak.
AAAI-specific review pressure
- Phase 1 reviewers need a fast reason to trust the evidence.
- The reproducibility checklist must match the experiment descriptions.
- AI for Social Impact and AI Alignment claims require stronger treatment of stakeholders, harms,
risk mitigation, and scope.
- New results usually cannot rescue the paper in rebuttal, so submit complete evidence upfront.
- The AI-assisted review pilot is non-decisional, but it may surface checklist mismatches; make
result provenance, seeds, data splits, and limits machine-readable enough that a human SPC/AC can
quickly audit them.
Pre-rebuttal freeze rule
Before submission, decide which experiments would be impossible to add later under AAAI's rebuttal
constraints: missing baselines, missing seeds, missing supplement files, or missing reproducibility
checklist answers. Treat those as pre-submission blockers, not rebuttal TODOs. The author
response can explain and clarify submitted evidence; it should not depend on new results, URLs, or
repaired supplementary files.
Evidence triage table
Because an AAAI reviewer from an adjacent subfield must trust your numbers quickly, classify each
experimental block by how much weight it can bear and what would strengthen it.
| Block |
Carries the claim when |
Reviewer doubt |
Cheap reinforcement |
| Headline benchmark |
beats tuned recent baselines |
"lucky seed" |
seeds, variance bars |
| Ablation |
isolates one mechanism |
"joint removal" |
single-factor toggles |
| Robustness |
holds across split/shift |
"one setting" |
extra split or perturbation |
| Human eval |
protocol is documented |
"rater bias" |
IRB note, inter-rater agreement |
Common AAAI experiment rejects
- Benchmark bump with no mechanism analysis, which a broad committee reads as engineering, not AI
insight.
- Baselines weaker than current open-source systems, so the comparison looks unfair.
- A Social-Impact or alignment claim with no stakeholder, harm, or risk-mitigation evidence.
- Results that rely on a closed API with no reproducible substitute for the checklist.
Worked vignette
A planning paper reports a single-seed win on one domain. Audit: the headline block "needs
robustness" and "needs variance", so the fix before the deadline is five seeds with confidence
intervals plus one extra IPC-style domain. Because new results cannot rescue this in rebuttal, the
team runs both before submission and aligns the checklist's seed answer to the supplement.
Output format
[Claim] <paper claim>
[Evidence status] sufficient / needs baseline / needs ablation / needs robustness / unclear
[Fairness issue] <compute, tuning, data, prompt, metric, human eval>
[Checklist dependency] <what checklist answer this supports>
[Pre-rebuttal blockers] <missing evidence that must be run before submission>
[Fast fix] <experiment or analysis feasible before deadline>
1---2name: aaai-experiments3description: Use when designing or auditing AAAI experiments for the broad-AI program committee, including baselines, ablations, statistical significance, robustness, human evaluation, AI-for-Social-Impact and alignment/safety evidence, compute and cost reporting, and reproducibility-checklist alignment for Phase-1 survival.4---56# AAAI Experiments78Use this before submission to ensure empirical evidence supports the AI contribution. AAAI9reviewers may come from adjacent AI subfields, so experiments must be interpretable beyond one10benchmark community.1112## Experiment audit1314- Map every experimental block to a claim in the introduction.15- Compare against strong, recent, and fairly tuned baselines.16- Include ablations that isolate mechanisms rather than removing multiple components at once.17- Report uncertainty, variance, and statistical tests when small differences matter.18- Test robustness to data split, prompt, seed, environment, user population, or distribution shift19 when relevant.20- For human evaluation, document task, instructions, annotator pool, quality control, aggregation,21 and ethics/IRB status.22- Report compute, hardware, data access, model size, and training/inference cost.2324## Claim-to-evidence ledger2526Build this table before adding new experiments. It keeps the AAAI evidence package aligned with27the main text and with the reproducibility checklist.2829| Manuscript claim | Required evidence | Phase-1 risk if missing | Checklist hook |30| --- | --- | --- | --- |31| New AI capability | benchmark + qualitative failure cases | broad reviewer sees only engineering | datasets, metrics, baselines |32| Better mechanism | single-factor ablations | gain looks like tuning luck | ablation and hyperparameter answers |33| Robust deployment | shift / seed / subgroup stress test | result seems brittle | variance, compute, environment |34| Social-impact or safety claim | stakeholder, harm, and misuse analysis | ethical claim looks asserted | ethics, limitations, data access |3536For each row, mark **ready / weak / missing** and name the fastest fix that can be run before the37supplementary-material deadline. Do not leave a claim in the abstract if its evidence row is weak.3839## AAAI-specific review pressure4041- Phase 1 reviewers need a fast reason to trust the evidence.42- The reproducibility checklist must match the experiment descriptions.43- AI for Social Impact and AI Alignment claims require stronger treatment of stakeholders, harms,44 risk mitigation, and scope.45- New results usually cannot rescue the paper in rebuttal, so submit complete evidence upfront.46- The AI-assisted review pilot is non-decisional, but it may surface checklist mismatches; make47 result provenance, seeds, data splits, and limits machine-readable enough that a human SPC/AC can48 quickly audit them.4950## Pre-rebuttal freeze rule5152Before submission, decide which experiments would be impossible to add later under AAAI's rebuttal53constraints: missing baselines, missing seeds, missing supplement files, or missing reproducibility54checklist answers. Treat those as **pre-submission blockers**, not rebuttal TODOs. The author55response can explain and clarify submitted evidence; it should not depend on new results, URLs, or56repaired supplementary files.5758## Evidence triage table5960Because an AAAI reviewer from an adjacent subfield must trust your numbers quickly, classify each61experimental block by how much weight it can bear and what would strengthen it.6263| Block | Carries the claim when | Reviewer doubt | Cheap reinforcement |64| --- | --- | --- | --- |65| Headline benchmark | beats tuned recent baselines | "lucky seed" | seeds, variance bars |66| Ablation | isolates one mechanism | "joint removal" | single-factor toggles |67| Robustness | holds across split/shift | "one setting" | extra split or perturbation |68| Human eval | protocol is documented | "rater bias" | IRB note, inter-rater agreement |6970## Common AAAI experiment rejects7172- Benchmark bump with no mechanism analysis, which a broad committee reads as engineering, not AI73 insight.74- Baselines weaker than current open-source systems, so the comparison looks unfair.75- A Social-Impact or alignment claim with no stakeholder, harm, or risk-mitigation evidence.76- Results that rely on a closed API with no reproducible substitute for the checklist.7778## Worked vignette7980A planning paper reports a single-seed win on one domain. Audit: the headline block "needs81robustness" and "needs variance", so the fix before the deadline is five seeds with confidence82intervals plus one extra IPC-style domain. Because new results cannot rescue this in rebuttal, the83team runs both before submission and aligns the checklist's seed answer to the supplement.8485## Output format8687```text88[Claim] <paper claim>89[Evidence status] sufficient / needs baseline / needs ablation / needs robustness / unclear90[Fairness issue] <compute, tuning, data, prompt, metric, human eval>91[Checklist dependency] <what checklist answer this supports>92[Pre-rebuttal blockers] <missing evidence that must be run before submission>93[Fast fix] <experiment or analysis feasible before deadline>94```
brycewang-stanford