ISSTA Artifact Evaluation
Use this to package evidence for ISSTA's artifact-evaluation track. ISSTA has a genuinely strong
artifact culture: a runnable tool and a shared benchmark are treated as normal, and the badges
carry weight. Reopen the current artifact call before packaging — the badge set and the archival
requirement are cycle-specific.
The ACM badge targets
| Badge |
What it certifies |
What you must ship |
| Artifacts Available |
The artifact is publicly, permanently retrievable |
A DOI-issuing deposit (Zenodo) — a GitHub link alone does not qualify |
| Artifacts Evaluated — Functional |
The artifact runs and does what the paper says |
A working entry point, dependencies, and a documented expected output |
| Artifacts Evaluated — Reusable |
Others can inspect, adapt, and build on it |
Clean structure, real documentation, and configurability beyond the paper's runs |
| Results Reproduced |
The paper's key results regenerate from the artifact |
Scripts that produce the paper's tables/figures within stated tolerance |
Available is about archival; Functional and Reusable are about engineering quality; Results
Reproduced is about matching the paper. They are earned independently — decide which you are
going for before you package.
Artifact plan
- Decide the badge set, then package to its bar. A Reusable badge needs documentation and structure
a Functional-only artifact can skip.
- Ship the tool with a single documented entry point and a container or environment file, so an
evaluator reaches a result without reconstructing your machine.
- Pin subjects and benchmark versions: the exact Defects4J revision, subject-program commit SHAs,
and any fuzzing seed corpora, archived rather than referenced by name.
- Provide a results-regeneration path: a script that takes logged runs to the paper's tables, so
"Results Reproduced" is a command, not an argument.
- Anonymize the review-time copy — repository owners, commit authors, container labels — because
ISSTA artifact review is double-anonymous alongside the paper.
- After acceptance, make the Zenodo deposit public and citable, and record its DOI for the
camera-ready badge display.
What ISSTA evaluators try first
- The README's quick-start, then the single command that produces one headline number. If that
fails on a clean machine, no amount of internal quality is visible.
- A small, fast subset that finishes in minutes, before any full multi-day fuzzing or
symbolic-execution campaign. Ship a "smoke" configuration explicitly.
- The mapping from a paper claim to the artifact output that supports it; an artifact whose outputs
cannot be tied back to Table N reads as unverifiable.
Handling long-running and non-deterministic tools
Testing and analysis artifacts often run for hours and vary between runs. Package for that reality:
artifact/
README.md # quick-start, smoke config, full config, expected outputs, runtime
Dockerfile # pinned toolchain and dependencies
subjects/ # pinned subject SHAs or a fetch script that pins them
run_smoke.sh # minutes: reproduces one representative row
run_full.sh # hours/days: reproduces all tables
results/ # logged raw outputs from the authors' runs
scripts/tables.py # regenerates paper tables from results/
State the run count and expected variance for non-deterministic results, and have the
regeneration script accept the evaluator's fresh runs as well as the shipped logs, so a partial
reproduction still lands on the paper's numbers within tolerance.
Calibration anchors
- Evaluators are time-boxed; assume they run the smoke config and skim the full one. Design so the
smoke path alone justifies Functional.
- Badge names, the Zenodo requirement, and any single-blind vs. double-blind detail vary by cycle;
verify against the current artifact call rather than a past year.
Output format
[Badge target] Available / Functional / Reusable / Results Reproduced
[Entry point] <command + smoke runtime>
[Pinned subjects] <benchmark version / SHAs archived?>
[Reproduction level] turnkey / scripted / descriptive / weak
[Anonymity risks] <owners/authors/labels/paths>
[Fixes before upload] <ordered list>
Source: brycewang-stanford/Awesome-Journal-Skills → ISSTA-Skills/skills/issta-artifact-evaluation/SKILL.md
1---2name: issta-artifact-evaluation3description: Use when packaging an ISSTA tool, benchmark, and results for the artifact-evaluation track, covering the ACM badges (Artifacts Available via Zenodo, Evaluated Functional and Reusable, Results Reproduced), the anonymous review-time copy, containerization, a runnable entry point, and what ISSTA artifact evaluators actually try first.4---567# ISSTA Artifact Evaluation89Use this to package evidence for ISSTA's artifact-evaluation track. ISSTA has a genuinely strong10artifact culture: a runnable tool and a shared benchmark are treated as normal, and the badges11carry weight. Reopen the current artifact call before packaging — the badge set and the archival12requirement are cycle-specific.1314## The ACM badge targets1516| Badge | What it certifies | What you must ship |17|---|---|---|18| Artifacts Available | The artifact is publicly, permanently retrievable | A DOI-issuing deposit (Zenodo) — a GitHub link alone does not qualify |19| Artifacts Evaluated — Functional | The artifact runs and does what the paper says | A working entry point, dependencies, and a documented expected output |20| Artifacts Evaluated — Reusable | Others can inspect, adapt, and build on it | Clean structure, real documentation, and configurability beyond the paper's runs |21| Results Reproduced | The paper's key results regenerate from the artifact | Scripts that produce the paper's tables/figures within stated tolerance |2223Available is about *archival*; Functional and Reusable are about *engineering quality*; Results24Reproduced is about *matching the paper*. They are earned independently — decide which you are25going for before you package.2627## Artifact plan2829- Decide the badge set, then package to its bar. A Reusable badge needs documentation and structure30 a Functional-only artifact can skip.31- Ship the tool with a single documented entry point and a container or environment file, so an32 evaluator reaches a result without reconstructing your machine.33- Pin subjects and benchmark versions: the exact Defects4J revision, subject-program commit SHAs,34 and any fuzzing seed corpora, archived rather than referenced by name.35- Provide a results-regeneration path: a script that takes logged runs to the paper's tables, so36 "Results Reproduced" is a command, not an argument.37- Anonymize the review-time copy — repository owners, commit authors, container labels — because38 ISSTA artifact review is double-anonymous alongside the paper.39- After acceptance, make the Zenodo deposit public and citable, and record its DOI for the40 camera-ready badge display.4142## What ISSTA evaluators try first4344- The README's quick-start, then the single command that produces one headline number. If that45 fails on a clean machine, no amount of internal quality is visible.46- A small, fast subset that finishes in minutes, before any full multi-day fuzzing or47 symbolic-execution campaign. Ship a "smoke" configuration explicitly.48- The mapping from a paper claim to the artifact output that supports it; an artifact whose outputs49 cannot be tied back to Table N reads as unverifiable.5051## Handling long-running and non-deterministic tools5253Testing and analysis artifacts often run for hours and vary between runs. Package for that reality:5455```text56artifact/57 README.md # quick-start, smoke config, full config, expected outputs, runtime58 Dockerfile # pinned toolchain and dependencies59 subjects/ # pinned subject SHAs or a fetch script that pins them60 run_smoke.sh # minutes: reproduces one representative row61 run_full.sh # hours/days: reproduces all tables62 results/ # logged raw outputs from the authors' runs63 scripts/tables.py # regenerates paper tables from results/64```6566State the run count and expected variance for non-deterministic results, and have the67regeneration script accept the evaluator's fresh runs as well as the shipped logs, so a partial68reproduction still lands on the paper's numbers within tolerance.6970## Calibration anchors7172- Evaluators are time-boxed; assume they run the smoke config and skim the full one. Design so the73 smoke path alone justifies Functional.74- Badge names, the Zenodo requirement, and any single-blind vs. double-blind detail vary by cycle;75 verify against the current artifact call rather than a past year.7677## Output format7879```text80[Badge target] Available / Functional / Reusable / Results Reproduced81[Entry point] <command + smoke runtime>82[Pinned subjects] <benchmark version / SHAs archived?>83[Reproduction level] turnkey / scripted / descriptive / weak84[Anonymity risks] <owners/authors/labels/paths>85[Fixes before upload] <ordered list>86```8788---8990**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `ISSTA-Skills/skills/issta-artifact-evaluation/SKILL.md`
thedixitjain