ICSME Experiments
Use this before submission when the empirical story is not yet locked. ICSME reviewers are
maintenance and evolution empiricists, and because the venue has no revision round, the
evaluation must be complete on submission — you cannot promise a missing analysis and add it later.
The organizing principle is evidence proportional to the claim, tested on real systems with
real change history, not on synthetic snapshots.
Evaluation audit
- Match evidence to the claim shape. A claim about maintenance effort needs effort or proxy
data defended as a proxy; a claim about change impact needs real change sets; a claim about
comprehension needs a human study; a claim about debt needs a debt measurement, not lines of
code. Accuracy against a convenient label is not evidence for a maintenance-practice claim.
- Use real, evolving subject systems, sampled by a stated criterion over a stated time window,
and list them in the artifact. A single snapshot cannot support an evolution claim.
- Pin mining provenance (see the code block): repository SHAs, extraction dates, inclusion/
exclusion criteria, and fork/duplicate/bot handling. Silent inclusion of forks or bot commits
skews every downstream evolution number.
- Choose fair baselines, including the strongest prior maintenance technique and a
simple-but-reasonable alternative, tuned with a documented, equal budget. An untuned baseline is a
scored weakness a rebuttal cannot fully repair.
- Report SE-standard statistics: effect sizes (not just p-values), confidence intervals,
appropriate tests, and corrections for multiple comparisons; say how many runs and what variance.
- Handle change-history confounds by design: survivorship (systems that died mid-history),
releases vs. commits as the unit of evolution, and time-window sensitivity.
- Hold qualitative work to method: coding schemes, inter-rater agreement, saturation, and an
audit trail — mixed-methods maintenance studies are native to ICSME.
Claim-to-evidence design table
| Maintenance/evolution claim |
Matching evidence |
Reject pattern avoided |
| "Technique reduces change impact" |
Real change sets from history, set-size deltas with CIs vs. tuned baseline |
"Evaluated on synthetic edits only" |
| "Refactoring preserves behaviour" |
Test outcomes / regression evidence across applications on real code |
"Assumed safe, never checked" |
| "Debt predicts future defects" |
Longitudinal link from debt measure to later real defects, with survivorship handled |
"Cross-sectional correlation claimed causal" |
| "Improves comprehension" |
Human study with tasks, accuracy/time, effect sizes |
"Tool complexity mistaken for understanding" |
| "Finding holds across systems" |
Diverse, dated sample + explicit external-validity limits |
"One ecosystem, claimed universal" |
| "The model adds the value" |
Ablation removing the model vs. lexical/heuristic features |
"Model's marginal contribution never isolated" |
Provenance floor for mining studies (the ICSME core)
[Corpus] pin repository SHAs; record extraction date and the time window studied;
archive the *extracted* dataset, not just the query
[Selection] state inclusion/exclusion criteria and resulting N; ship the filtering script
[Hygiene] document fork/duplicate/bot handling; report how merges and renames were resolved
[Unit] state whether the unit is commit, PR, release, or file, and defend it for the claim
[History] address survivorship and left-truncation of the change history explicitly
Contamination-aware LLM evaluation
When an LLM is in the loop (increasingly common for comprehension, summarization, and repair):
[Contamination] are subject systems plausibly in the model's training data? report model cutoff
vs. project/commit dates; prefer post-cutoff or held-out systems
[Determinism] fix temperature/seed where possible; report the sampling settings
[Caching] store raw prompts and raw responses in the artifact; a live-API study re-samples
rather than reproduces
[Versioning] record exact model identifiers and access dates; models drift under a name
[Ablation] isolate the model's marginal value against a non-LLM maintenance baseline
Vignette: evaluating a change-impact technique
Suppose the paper claims a technique predicts change-impact sets more precisely than a prior tool.
The matching plan: draw real change sets from dated release histories of several evolving systems;
run both tools with an equal, documented budget; report precision/recall with confidence intervals
and an effect size; manually inspect a sample of predicted impacts for spurious inclusion; handle
survivorship by stating which systems' histories are truncated; and bound external validity
(languages, domains) as a threat — every number traceable to a logged run in the artifact.
Statistical reporting floor
- Effect sizes and confidence intervals for every quantitative comparison; say what the intervals
represent.
- Number of runs and the source of variance for any stochastic component.
- The compute and mining time actually consumed, not vague feasibility language.
Output format
[Evaluation readiness] strong / adequate / weak
[Claim -> evidence map] <claim: systems/history-window/metric/statistic>
[Mining provenance] <SHAs / extraction date / selection / fork-bot hygiene handled? yes/no>
[Baseline fairness] <baseline -> tuned? equal budget? documented?>
[Confounds by design] <survivorship / unit-of-analysis / contamination -> how bounded>
[Decision-critical gap] <the one thing that must be complete before submission — no revision round>
1---2name: icsme-experiments3description: Use when designing or auditing IEEE ICSME empirical evaluations, covering real evolving subject systems, mining-software-repositories provenance, fair baselines, SE-standard statistics and effect sizes, change-history and survivorship confounds, qualitative rigor, contamination-aware LLM ablations, and matching evidence to the shape of each maintenance/evolution claim.4---56# ICSME Experiments78Use this before submission when the empirical story is not yet locked. ICSME reviewers are9maintenance and evolution empiricists, and because the venue has **no revision round**, the10evaluation must be complete on submission — you cannot promise a missing analysis and add it later.11The organizing principle is **evidence proportional to the claim**, tested on **real systems with12real change history**, not on synthetic snapshots.1314## Evaluation audit1516- **Match evidence to the claim shape.** A claim about *maintenance effort* needs effort or proxy17 data defended as a proxy; a claim about *change impact* needs real change sets; a claim about18 *comprehension* needs a human study; a claim about *debt* needs a debt measurement, not lines of19 code. Accuracy against a convenient label is not evidence for a maintenance-practice claim.20- **Use real, evolving subject systems,** sampled by a stated criterion over a stated time window,21 and list them in the artifact. A single snapshot cannot support an evolution claim.22- **Pin mining provenance** (see the code block): repository SHAs, extraction dates, inclusion/23 exclusion criteria, and fork/duplicate/bot handling. Silent inclusion of forks or bot commits24 skews every downstream evolution number.25- **Choose fair baselines,** including the strongest prior maintenance technique and a26 simple-but-reasonable alternative, tuned with a documented, equal budget. An untuned baseline is a27 scored weakness a rebuttal cannot fully repair.28- **Report SE-standard statistics:** effect sizes (not just p-values), confidence intervals,29 appropriate tests, and corrections for multiple comparisons; say how many runs and what variance.30- **Handle change-history confounds by design:** survivorship (systems that died mid-history),31 releases vs. commits as the unit of evolution, and time-window sensitivity.32- **Hold qualitative work to method:** coding schemes, inter-rater agreement, saturation, and an33 audit trail — mixed-methods maintenance studies are native to ICSME.3435## Claim-to-evidence design table3637| Maintenance/evolution claim | Matching evidence | Reject pattern avoided |38|---|---|---|39| "Technique reduces change impact" | Real change sets from history, set-size deltas with CIs vs. tuned baseline | "Evaluated on synthetic edits only" |40| "Refactoring preserves behaviour" | Test outcomes / regression evidence across applications on real code | "Assumed safe, never checked" |41| "Debt predicts future defects" | Longitudinal link from debt measure to later real defects, with survivorship handled | "Cross-sectional correlation claimed causal" |42| "Improves comprehension" | Human study with tasks, accuracy/time, effect sizes | "Tool complexity mistaken for understanding" |43| "Finding holds across systems" | Diverse, dated sample + explicit external-validity limits | "One ecosystem, claimed universal" |44| "The model adds the value" | Ablation removing the model vs. lexical/heuristic features | "Model's marginal contribution never isolated" |4546## Provenance floor for mining studies (the ICSME core)4748```text49[Corpus] pin repository SHAs; record extraction date and the time window studied;50 archive the *extracted* dataset, not just the query51[Selection] state inclusion/exclusion criteria and resulting N; ship the filtering script52[Hygiene] document fork/duplicate/bot handling; report how merges and renames were resolved53[Unit] state whether the unit is commit, PR, release, or file, and defend it for the claim54[History] address survivorship and left-truncation of the change history explicitly55```5657## Contamination-aware LLM evaluation5859When an LLM is in the loop (increasingly common for comprehension, summarization, and repair):6061```text62[Contamination] are subject systems plausibly in the model's training data? report model cutoff63 vs. project/commit dates; prefer post-cutoff or held-out systems64[Determinism] fix temperature/seed where possible; report the sampling settings65[Caching] store raw prompts and raw responses in the artifact; a live-API study re-samples66 rather than reproduces67[Versioning] record exact model identifiers and access dates; models drift under a name68[Ablation] isolate the model's marginal value against a non-LLM maintenance baseline69```7071## Vignette: evaluating a change-impact technique7273Suppose the paper claims a technique predicts change-impact sets more precisely than a prior tool.74The matching plan: draw real change sets from dated release histories of several evolving systems;75run both tools with an equal, documented budget; report precision/recall with confidence intervals76and an effect size; manually inspect a sample of predicted impacts for spurious inclusion; handle77survivorship by stating which systems' histories are truncated; and bound external validity78(languages, domains) as a threat — every number traceable to a logged run in the artifact.7980## Statistical reporting floor8182- Effect sizes and confidence intervals for every quantitative comparison; say what the intervals83 represent.84- Number of runs and the source of variance for any stochastic component.85- The compute and mining time actually consumed, not vague feasibility language.8687## Output format8889```text90[Evaluation readiness] strong / adequate / weak91[Claim -> evidence map] <claim: systems/history-window/metric/statistic>92[Mining provenance] <SHAs / extraction date / selection / fork-bot hygiene handled? yes/no>93[Baseline fairness] <baseline -> tuned? equal budget? documented?>94[Confounds by design] <survivorship / unit-of-analysis / contamination -> how bounded>95[Decision-critical gap] <the one thing that must be complete before submission — no revision round>96```
brycewang-stanford