ISSTA Experiments
Use this before submission when the evaluation is not yet locked. ISSTA experiments earn or lose the
paper on the evaluation criterion, and the reviewer pool knows the standard subjects, baselines, and
statistics — so design the study to answer the exact claim, on subjects a reviewer recognizes.
Experiment audit
- Match each claim to its evidence: a bug-finding claim needs a benchmark with ground truth, a
coverage claim needs a measurement protocol, a scalability claim needs a size sweep.
- Use real subjects and established benchmarks where they exist — Defects4J for Java faults,
real-world CVEs for security bugs, standard fuzzing corpora — so results are comparable to prior
work, not to a private subject set.
- Configure baselines fairly and at an equal budget: same time, same seeds, same subjects. A
baseline throttled to lose is the fastest way to lose a reviewer's trust.
- Compare with proper statistics: non-parametric tests (e.g. Mann-Whitney U) and an effect size
(e.g. Vargha-Delaney Â₁₂) rather than a single run, because testing and analysis results are
stochastic and rarely normal.
- Report the stochastic protocol: seeds, timeout budgets, iteration counts, number of repeated runs,
and hardware. State whether a table is a mean over runs and give the spread.
- Audit for the usual confounds: subject selection bias, overfitting to the benchmark, counting
duplicate crashes as distinct bugs, and mismatches between what the metric measures and what the
claim asserts.
What experiments are for at this venue
- ISSTA experiments exist to show a technique works on software that matters, evaluated fairly.
One well-designed study on an established benchmark outweighs five extra ad-hoc subjects.
- The strongest evaluations separate finding from being right: a repair that passes tests is not
a correct repair; a crash is not automatically a distinct bug. Report the gap between the easy
metric and the property you care about.
- Reviewers check that the subjects and budget make the comparison meaningful — a fuzzer compared for
one hour says little about a claim over 24-hour campaigns.
Claim-to-evidence design table
| Claim |
Matching evaluation |
Reject pattern avoided |
| "Finds more true bugs" |
Established benchmark with ground-truth labels; dedup by root cause |
"Bugs counted by distinct crashes, not distinct faults" |
| "Higher coverage than baseline X" |
Equal budget, same subjects, same instrumentation |
"Baseline run in a weaker configuration" |
| "Scales to large programs" |
Size sweep with wall-clock and memory reported |
"Scalability asserted, never measured" |
| "Improvement is real, not noise" |
Repeated runs + non-parametric test + effect size |
"Single run presented as representative" |
Vignette: evaluating a test-generation tool
A paper claims a new generator achieves higher fault-detection than an existing one. The plan:
run both on Defects4J at an equal per-subject time budget, repeat each configuration enough times to
estimate variance, count detected faults (not generated tests), compare with a non-parametric test
and report Â₁₂, and disclose the subjects where the tool underperforms rather than dropping them.
subjects Defects4J (pinned revision), N faults
budget equal wall-clock per subject, both tools
runs R repeats per subject; report mean and spread
metric faults detected (ground truth), not tests generated
statistic Mann-Whitney U + Vargha-Delaney A12
honesty report subjects where the tool loses
Statistical reporting floor
- Repeated runs and seeds for every stochastic result; captions must say what the numbers summarize.
- Report the compute actually consumed, not vague feasibility language.
Output format
[Evaluation readiness] strong / adequate / weak
[Claim -> evidence map] <claim: benchmark/metric/statistic>
[Baseline fairness] equal-budget / configured-to-lose / unclear
[Statistical gaps] <runs / test / effect size / variance>
[Subject/benchmark gaps] <established benchmark used? subjects pinned?>
[Decision-critical next run] <one experiment>
1---2name: issta-experiments3description: Use when designing or auditing ISSTA experiments, covering real subject programs and benchmarks like Defects4J, fair tool-baseline configuration, bug-finding and coverage metrics, non-parametric comparison with effect sizes, equal-budget protocols, repeated runs, and matching evidence to the claim being made.4---56# ISSTA Experiments78Use this before submission when the evaluation is not yet locked. ISSTA experiments earn or lose the9paper on the evaluation criterion, and the reviewer pool knows the standard subjects, baselines, and10statistics — so design the study to answer the exact claim, on subjects a reviewer recognizes.1112## Experiment audit1314- Match each claim to its evidence: a bug-finding claim needs a benchmark with ground truth, a15 coverage claim needs a measurement protocol, a scalability claim needs a size sweep.16- Use real subjects and established benchmarks where they exist — Defects4J for Java faults,17 real-world CVEs for security bugs, standard fuzzing corpora — so results are comparable to prior18 work, not to a private subject set.19- Configure baselines fairly and at an **equal budget**: same time, same seeds, same subjects. A20 baseline throttled to lose is the fastest way to lose a reviewer's trust.21- Compare with proper statistics: non-parametric tests (e.g. Mann-Whitney U) and an effect size22 (e.g. Vargha-Delaney Â₁₂) rather than a single run, because testing and analysis results are23 stochastic and rarely normal.24- Report the stochastic protocol: seeds, timeout budgets, iteration counts, number of repeated runs,25 and hardware. State whether a table is a mean over runs and give the spread.26- Audit for the usual confounds: subject selection bias, overfitting to the benchmark, counting27 duplicate crashes as distinct bugs, and mismatches between what the metric measures and what the28 claim asserts.2930## What experiments are for at this venue3132- ISSTA experiments exist to show a technique *works on software that matters*, evaluated fairly.33 One well-designed study on an established benchmark outweighs five extra ad-hoc subjects.34- The strongest evaluations separate *finding* from *being right*: a repair that passes tests is not35 a correct repair; a crash is not automatically a distinct bug. Report the gap between the easy36 metric and the property you care about.37- Reviewers check that the subjects and budget make the comparison meaningful — a fuzzer compared for38 one hour says little about a claim over 24-hour campaigns.3940## Claim-to-evidence design table4142| Claim | Matching evaluation | Reject pattern avoided |43|---|---|---|44| "Finds more true bugs" | Established benchmark with ground-truth labels; dedup by root cause | "Bugs counted by distinct crashes, not distinct faults" |45| "Higher coverage than baseline X" | Equal budget, same subjects, same instrumentation | "Baseline run in a weaker configuration" |46| "Scales to large programs" | Size sweep with wall-clock and memory reported | "Scalability asserted, never measured" |47| "Improvement is real, not noise" | Repeated runs + non-parametric test + effect size | "Single run presented as representative" |4849## Vignette: evaluating a test-generation tool5051A paper claims a new generator achieves higher fault-detection than an existing one. The plan:52run both on Defects4J at an equal per-subject time budget, repeat each configuration enough times to53estimate variance, count *detected faults* (not generated tests), compare with a non-parametric test54and report Â₁₂, and disclose the subjects where the tool underperforms rather than dropping them.5556```text57subjects Defects4J (pinned revision), N faults58budget equal wall-clock per subject, both tools59runs R repeats per subject; report mean and spread60metric faults detected (ground truth), not tests generated61statistic Mann-Whitney U + Vargha-Delaney A1262honesty report subjects where the tool loses63```6465## Statistical reporting floor6667- Repeated runs and seeds for every stochastic result; captions must say what the numbers summarize.68- Report the compute actually consumed, not vague feasibility language.6970## Output format7172```text73[Evaluation readiness] strong / adequate / weak74[Claim -> evidence map] <claim: benchmark/metric/statistic>75[Baseline fairness] equal-budget / configured-to-lose / unclear76[Statistical gaps] <runs / test / effect size / variance>77[Subject/benchmark gaps] <established benchmark used? subjects pinned?>78[Decision-critical next run] <one experiment>79```
brycewang-stanford