IJCAI Experiments
Use this before submission when the experimental story is not yet locked. IJCAI reviewers
can score novelty, correctness, clarity, significance, impact, presentation, ethics, and
reproducibility.
Experiment audit
- Map each major claim to a table, figure, theorem, ablation, proof, or qualitative analysis.
- Include strong, current, and properly tuned baselines; explain any missing baseline before
reviewers ask.
- Report dataset splits, preprocessing, metrics, search ranges, final hyperparameters,
selection criteria, random seeds or repeats, and compute infrastructure.
- Add ablations for the core mechanism, not just peripheral architecture choices.
- Use uncertainty estimates, paired tests, confidence intervals, or repeated runs when small
differences could change the conclusion.
- For sensitive data or human-facing systems, document privacy, consent, copyright, safety,
fairness, misuse, and deployment limits.
- Keep enough details in the main paper for credible reproduction even if reviewers ignore
the supplementary material.
What IJCAI reviewers score the evidence on
IJCAI draws reviewers from symbolic AI, search, planning, constraint satisfaction, KR,
multi-agent systems, game theory, ML, NLP, and vision, so the experimental section must read
across subcommunities. Calibrate evidence to the claim type rather than copying an ML-only
template.
| Contribution type |
Decisive evidence |
Common reject trigger |
| Search / planning |
Coverage, anytime quality, expansion counts, time/memory cutoffs, per-domain breakdown |
Single suite, no domain table, missing strong planner baseline |
| Constraint / SAT |
Cactus plots, instances within timeout, solver versions |
No virtual-best comparison |
| Multi-agent / game theory |
Welfare/equilibrium metrics, agent-count scaling, seeds |
Claims hold at one population size only |
| Learning method |
Strong current baselines, core-mechanism ablations, variance |
Cherry-picked seeds, weak baselines |
| Theory-plus-experiment |
Experiments confirming the proven bound |
Empirics outside the theorem's regime |
Worked vignette: a heuristic-search paper
A submission proposes a learned heuristic for cost-optimal classical planning and reports a
single aggregate "12% fewer expansions" number. Apply the decision rules:
- Evidence and baseline: replace the single mean with a per-domain coverage and expansion
table, and add a strong admissible-heuristic baseline that shows optimality is preserved.
- Core ablation: isolate the learned component from the search framework so the gain is not
credited to engineering, and report multiple training seeds since the heuristic is stochastic.
- Compute: state the planner, time and memory limits, and machine, since coverage is
meaningless without a stated timeout.
Reviewer pushback and the venue-specific fix
- "Only one benchmark family." Add a second problem class or justify the scope; an IJCAI
cross-section reviewer distrusts single-suite claims.
- "Baseline is outdated." Cite and run a current top method; the broad PC notices stale
comparisons.
- "Gains are within noise." Provide repeats, paired tests, or confidence intervals before the
response, since no new results may be added later.
Output format
[Experiment readiness] strong / adequate / weak
[Claim -> evidence map] <claim: section/table/figure>
[Missing baseline or ablation] <item>
[Reproducibility gaps] <hyperparameters/seeds/compute/data/code>
[Decision-critical next run] <one experiment>
1---2name: ijcai-experiments3description: Use when designing or auditing IJCAI or IJCAI-ECAI experiments, baselines, ablations, statistical evidence, hyperparameter reporting, compute descriptions, dataset handling, ethics risks, and reproducibility evidence for AI papers.4---56# IJCAI Experiments78Use this before submission when the experimental story is not yet locked. IJCAI reviewers9can score novelty, correctness, clarity, significance, impact, presentation, ethics, and10reproducibility.1112## Experiment audit1314- Map each major claim to a table, figure, theorem, ablation, proof, or qualitative analysis.15- Include strong, current, and properly tuned baselines; explain any missing baseline before16 reviewers ask.17- Report dataset splits, preprocessing, metrics, search ranges, final hyperparameters,18 selection criteria, random seeds or repeats, and compute infrastructure.19- Add ablations for the core mechanism, not just peripheral architecture choices.20- Use uncertainty estimates, paired tests, confidence intervals, or repeated runs when small21 differences could change the conclusion.22- For sensitive data or human-facing systems, document privacy, consent, copyright, safety,23 fairness, misuse, and deployment limits.24- Keep enough details in the main paper for credible reproduction even if reviewers ignore25 the supplementary material.2627## What IJCAI reviewers score the evidence on2829IJCAI draws reviewers from symbolic AI, search, planning, constraint satisfaction, KR,30multi-agent systems, game theory, ML, NLP, and vision, so the experimental section must read31across subcommunities. Calibrate evidence to the claim type rather than copying an ML-only32template.3334| Contribution type | Decisive evidence | Common reject trigger |35| --- | --- | --- |36| Search / planning | Coverage, anytime quality, expansion counts, time/memory cutoffs, per-domain breakdown | Single suite, no domain table, missing strong planner baseline |37| Constraint / SAT | Cactus plots, instances within timeout, solver versions | No virtual-best comparison |38| Multi-agent / game theory | Welfare/equilibrium metrics, agent-count scaling, seeds | Claims hold at one population size only |39| Learning method | Strong current baselines, core-mechanism ablations, variance | Cherry-picked seeds, weak baselines |40| Theory-plus-experiment | Experiments confirming the proven bound | Empirics outside the theorem's regime |4142## Worked vignette: a heuristic-search paper4344A submission proposes a learned heuristic for cost-optimal classical planning and reports a45single aggregate "12% fewer expansions" number. Apply the decision rules:46471. Evidence and baseline: replace the single mean with a per-domain coverage and expansion48 table, and add a strong admissible-heuristic baseline that shows optimality is preserved.492. Core ablation: isolate the learned component from the search framework so the gain is not50 credited to engineering, and report multiple training seeds since the heuristic is stochastic.513. Compute: state the planner, time and memory limits, and machine, since coverage is52 meaningless without a stated timeout.5354## Reviewer pushback and the venue-specific fix5556- "Only one benchmark family." Add a second problem class or justify the scope; an IJCAI57 cross-section reviewer distrusts single-suite claims.58- "Baseline is outdated." Cite and run a current top method; the broad PC notices stale59 comparisons.60- "Gains are within noise." Provide repeats, paired tests, or confidence intervals before the61 response, since no new results may be added later.6263## Output format6465```text66[Experiment readiness] strong / adequate / weak67[Claim -> evidence map] <claim: section/table/figure>68[Missing baseline or ablation] <item>69[Reproducibility gaps] <hyperparameters/seeds/compute/data/code>70[Decision-critical next run] <one experiment>71```
brycewang-stanford