Ieeesp Reproducibility

IEEE S&P Reproducibility

Use this while the evidence is being produced — months before ieeesp-artifact-evaluation packages it. Security results decay in ways ML results do not: targets get patched, infrastructure changes, and an attack that "worked in March" can be unreproducible by review time. Reproducibility at S&P is first about freezing the world you measured.

Freeze the target, not just the code

Record these in the paper, not only in lab notes — a reviewer asking "does this still work after 's April update?" is a reproducibility question about the world, and the answer is a documented snapshot date.

Determinism ledger for probabilistic evidence

Attacks with success probabilities, fuzzing campaigns, and randomized defenses need trial discipline:

• Report trials, not anecdotes: success rate over n runs with n stated, plus dispersion — a "9/10 successful" exploit and a "measured once" exploit are different claims.
• Fuzzing comparisons follow the field's known pitfalls: equal CPU-time budgets, multiple campaigns, identical seed corpora — state all three.
• Randomized defenses (ASLR-like, moving-target): evaluate across the randomness, not one lucky layout.
• Timing measurements: report the noise floor and the machine's quiescence conditions (isolated cores, frequency pinning) or the numbers will not transfer.

Reproducibility ledger (one row per experiment in the paper):
  exp_id | claim it supports | target snapshot (ver/date/hw) |
  trials & seeds | dispersion reported? | rerun cost (time/hw/$) |
  rerunnable by outsider? (yes / gated / world-dependent)

The last column becomes the availability statement and the honest badge target later.

When ethics limits release, say exactly what and why

S&P reviewers accept withheld material when the reasoning is specific:

• "Exploit for CVE-pending issue withheld until the fix ships; released to reviewers via the chairs on request" beats a silent gap.
• User-derived datasets: describe schema and collection so others can re-collect ethically, even when raw data cannot ship.
• Never fabricate openness — a promised-but-empty repository found during review or after publication is a reputation event, and at this venue reviewers do check.

Cheap habits that pay at rebuttal time

• One env.lock per experiment directory: container digest, package list, kernel and microcode versions, dumped automatically by the run script.
• Raw outputs archived before aggregation; the rebuttal question is always about a number two steps upstream of the figure.
• A regenerate_figures.sh that goes from archived raw data to every figure — this is also the artifact-evaluation core later.
• Date-stamped disclosure and measurement logs, because ethics questions in review are answered with timelines (ieeesp-author-response).

What this venue does not require

Keep effort calibrated: S&P has no submission-time reproducibility checklist in the verified 2026/2027 materials (待核实 each cycle), artifact evaluation is post-acceptance and optional, and appendices are explicitly not guaranteed reader attention. The reproducibility work above is therefore aimed at three audiences in order: your own rebuttal, the shepherd, and the AE committee — not at a submission-form requirement.

Output format

[Ledger status] <n>/<total> experiments with snapshot + trials + dispersion
[World-dependence] <which results cannot be re-run by anyone, ever — flagged in text?>
[Release plan] open / gated (reason) / withheld (reason) — per component
[Rebuttal readiness] raw data archived ✓/✗ · env locks ✓/✗ · figure regen ✓/✗
[Gaps to close before registration week] <ordered list>

Source: brycewang-stanford/Awesome-Journal-Skills → IEEE-SP-Skills/skills/ieeesp-reproducibility/SKILL.md