ASE Experiments
Match the evidence to the automation's claim. ASE evaluations are judged on whether a tool or
technique actually does what it claims on real subjects, compared fairly against the closest
runnable automation. This is the axis reviewers weight most, and the one that most often becomes a
Revision criterion.
Start from the claim shape
Different automations demand different evidence:
| Automation claim |
Evidence that matches |
Common failure |
| Detection (bugs, smells, vulnerabilities) |
Precision/recall/F on real defects with a defined ground truth |
Synthetic-only defects; unclear ground truth |
| Generation / synthesis (tests, code, patches) |
Validity of the produced artifact (compiles, passes, holds the property) |
Similarity-to-reference proxy instead of validity |
| Repair |
Verified behavior change: re-run + oracle; assertion/spec preservation |
"Plausible patch" without an overfitting check |
| Localization / ranking |
Rank-based effectiveness on real faults vs. alternatives |
Cherry-picked programs; one metric only |
| Scalability / performance |
Real-system sizes, wall-clock with a fair config |
Toy inputs; unequal baseline budget |
Real subject systems
- Use real software — open-source projects, real bug/defect datasets, real CI logs — not toy
programs you constructed to make the tool look good.
- Report subject provenance: names, versions/commit SHAs, sizes, and the extraction date.
Reviewers reproduce from this.
- Justify subject selection and disclose exclusions; self-selected subjects are the classic
external-validity threat.
Fair, runnable tool baselines
- Compare against the closest runnable automation, configured at an equal, documented budget
(time, iterations, tuning, seeds). ASE reviewers routinely rerun or scrutinize baselines.
- Pin baseline versions/commits and note reimplementation vs. original.
- If no tool baseline exists, construct a defensible non-trivial baseline (a static rewrite, a
random or heuristic variant) rather than comparing only to "nothing."
Ablations that isolate the automation
If a learned or LLM component is involved, run an ablation that removes it and keeps the rest,
so the marginal value of the design is visible. This is what defeats the "the model did it, not
your technique" objection and keeps the paper ASE-shaped rather than ML-shaped.
Oracles and correctness
- State the oracle explicitly: how do you know a generated test is meaningful, or a repair is
correct? Re-execution, differential testing, formal checks, or human audit — name it.
- For repair/synthesis, guard against overfitting to the evaluation oracle (e.g., patches that
pass the given tests but break behavior): report a held-out or manual correctness check.
Statistics and effect sizes
- Report effect sizes and dispersion (confidence intervals, non-parametric tests where
appropriate), not just point estimates or a single accuracy number.
- For randomized techniques (search-based, sampling, LLM temperature > 0), report repeated runs
with variance and fix/seed the randomness for the artifact.
Contamination-aware LLM handling
- Record model identifiers and dates; a model updated between runs invalidates comparisons.
- Consider training-data contamination: benchmarks the model may have seen inflate results —
report on held-out or post-cutoff subjects where feasible, and say so.
- Cache raw model outputs so the artifact reproduces rather than re-samples a live API.
Mining and dataset provenance
- Pin repository SHAs, the corpus extraction date, query/filter criteria, and any labeling
protocol with inter-rater agreement for manually coded data.
- Version the dataset and describe how to regenerate it; a package that needs live scraping
re-samples a moving target.
Evaluation audit checklist
[Claim-evidence] each claim -> a matching metric on real subjects (not a proxy)
[Subjects] real, provenance-pinned, selection justified, exclusions disclosed
[Baselines] closest runnable tool, version pinned, equal documented budget
[Ablation] learned/LLM component isolated; marginal value of the design shown
[Oracle] correctness defined; overfitting-to-oracle checked
[Stats] effect sizes + dispersion; repeated runs for randomized methods
[LLM] model IDs/dates recorded; contamination considered; outputs cached
[Repro] provenance pinned; dataset/tool versioned for the artifact
Output format
[Automation claim] detection / generation / repair / localization / scalability
[Evidence match] metric(s) that fit the claim, on real subjects
[Baseline fairness] closest tool, budget parity, versions
[Ablation + oracle] learned-component ablation present; correctness oracle stated
[Threats] subject selection / oracle validity / baseline fairness / contamination — bounded how?
Source: brycewang-stanford/Awesome-Journal-Skills → ASE-Skills/skills/ase-experiments/SKILL.md
1---2name: ase-experiments3description: Use when designing or auditing the evaluation of an ASE (IEEE/ACM Automated Software Engineering) paper, covering real subject systems, fair runnable tool baselines, task-matched effectiveness metrics, ablations that isolate a learned component, oracle and correctness validation, contamination-aware LLM handling, and provenance for mining.4---567# ASE Experiments89Match the evidence to the automation's claim. ASE evaluations are judged on whether a **tool or10technique actually does what it claims on real subjects**, compared **fairly** against the closest11runnable automation. This is the axis reviewers weight most, and the one that most often becomes a12Revision criterion.1314## Start from the claim shape1516Different automations demand different evidence:1718| Automation claim | Evidence that matches | Common failure |19|---|---|---|20| Detection (bugs, smells, vulnerabilities) | Precision/recall/F on real defects with a defined ground truth | Synthetic-only defects; unclear ground truth |21| Generation / synthesis (tests, code, patches) | Validity of the produced artifact (compiles, passes, holds the property) | Similarity-to-reference proxy instead of validity |22| Repair | Verified behavior change: re-run + oracle; assertion/spec preservation | "Plausible patch" without an overfitting check |23| Localization / ranking | Rank-based effectiveness on real faults vs. alternatives | Cherry-picked programs; one metric only |24| Scalability / performance | Real-system sizes, wall-clock with a fair config | Toy inputs; unequal baseline budget |2526## Real subject systems2728- Use **real software** — open-source projects, real bug/defect datasets, real CI logs — not toy29 programs you constructed to make the tool look good.30- Report subject **provenance**: names, versions/commit SHAs, sizes, and the extraction date.31 Reviewers reproduce from this.32- Justify subject **selection** and disclose exclusions; self-selected subjects are the classic33 external-validity threat.3435## Fair, runnable tool baselines3637- Compare against the **closest runnable automation**, configured at an **equal, documented budget**38 (time, iterations, tuning, seeds). ASE reviewers routinely rerun or scrutinize baselines.39- Pin baseline **versions/commits** and note reimplementation vs. original.40- If no tool baseline exists, construct a defensible non-trivial baseline (a static rewrite, a41 random or heuristic variant) rather than comparing only to "nothing."4243## Ablations that isolate the automation4445If a learned or LLM component is involved, run an **ablation** that removes it and keeps the rest,46so the marginal value of the *design* is visible. This is what defeats the "the model did it, not47your technique" objection and keeps the paper ASE-shaped rather than ML-shaped.4849## Oracles and correctness5051- State the **oracle** explicitly: how do you know a generated test is meaningful, or a repair is52 correct? Re-execution, differential testing, formal checks, or human audit — name it.53- For repair/synthesis, guard against **overfitting** to the evaluation oracle (e.g., patches that54 pass the given tests but break behavior): report a held-out or manual correctness check.5556## Statistics and effect sizes5758- Report **effect sizes and dispersion** (confidence intervals, non-parametric tests where59 appropriate), not just point estimates or a single accuracy number.60- For randomized techniques (search-based, sampling, LLM temperature > 0), report **repeated runs**61 with variance and fix/seed the randomness for the artifact.6263## Contamination-aware LLM handling6465- Record **model identifiers and dates**; a model updated between runs invalidates comparisons.66- Consider **training-data contamination**: benchmarks the model may have seen inflate results —67 report on held-out or post-cutoff subjects where feasible, and say so.68- **Cache raw model outputs** so the artifact reproduces rather than re-samples a live API.6970## Mining and dataset provenance7172- Pin repository SHAs, the corpus extraction date, query/filter criteria, and any labeling73 protocol with inter-rater agreement for manually coded data.74- Version the dataset and describe how to regenerate it; a package that needs live scraping75 re-samples a moving target.7677## Evaluation audit checklist7879```text80[Claim-evidence] each claim -> a matching metric on real subjects (not a proxy)81[Subjects] real, provenance-pinned, selection justified, exclusions disclosed82[Baselines] closest runnable tool, version pinned, equal documented budget83[Ablation] learned/LLM component isolated; marginal value of the design shown84[Oracle] correctness defined; overfitting-to-oracle checked85[Stats] effect sizes + dispersion; repeated runs for randomized methods86[LLM] model IDs/dates recorded; contamination considered; outputs cached87[Repro] provenance pinned; dataset/tool versioned for the artifact88```8990## Output format9192```text93[Automation claim] detection / generation / repair / localization / scalability94[Evidence match] metric(s) that fit the claim, on real subjects95[Baseline fairness] closest tool, budget parity, versions96[Ablation + oracle] learned-component ablation present; correctness oracle stated97[Threats] subject selection / oracle validity / baseline fairness / contamination — bounded how?98```99100---101102**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `ASE-Skills/skills/ase-experiments/SKILL.md`
thedixitjain