CCS Artifact Evaluation
Use this for artifact packaging around CCS. CCS runs an optional artifact-evaluation process
after acceptance; passing artifacts earn ACM badges that appear on the paper's first page.
Reopen the current call for artifacts for the exact badge set, submission form, and deadlines.
The ACM badge ladder
| Badge |
What it certifies |
What the committee does |
| Artifacts Available |
The artifact is publicly archived with a stable identifier |
Confirms the artifact is retrievable and permanent |
| Artifacts Evaluated - Functional |
The artifact runs and does what the paper says |
Executes it against the documented steps |
| Artifacts Evaluated - Reusable |
It exceeds functional quality and others can reuse it |
Judges structure, documentation, and reusability |
| Results Reproduced |
The main paper results are independently reproduced |
Reruns experiments and checks they support the claims |
Available is about archival permanence; Functional and Reusable are about quality; Results
Reproduced is the highest bar and requires the committee to regenerate your headline numbers.
Artifact plan
- Decide which claims the artifact must support: the exploit, the defense overhead, the
measurement pipeline, or the protocol implementation.
- Make the security claim turnkey: one documented command per headline result, with expected
output, runtime, and the hardware assumed.
- Anonymize nothing at this stage — artifact evaluation is post-acceptance and single-blind or
open — but do pin versions, dependencies, and a container so it runs on a clean machine.
- For dangerous artifacts (working exploits, malware, live attack tooling), gate access,
document safe-handling, and follow the responsible-disclosure posture from the paper.
- Justify anything withheld: licensing, disclosure embargo, subject safety, or premature-release
risk, and offer partial, synthetic, or redacted substitutes that still let evaluators check
the method.
What security evaluators open first
| Claim type |
First artifact inspected |
Common failure caught |
| Exploit against real software |
The PoC and its target build instructions |
Target version unspecified; PoC fails to build |
| Defense with overhead numbers |
The benchmark runner and instrumented build |
Overhead cannot be reproduced; workload undocumented |
| Measurement study |
The collection and classification scripts |
Classification rule differs from the paper's prose |
| Cryptographic implementation |
Test vectors and the reference build |
No test vectors; outputs cannot be checked |
Worked vignette: packaging a fuzzing-tool artifact
A hypothetical accepted paper presents a new fuzzer that found N bugs. For Reusable and
Results Reproduced: ship a container pinning the fuzzer, the target corpus, and the seed set;
document one command to reproduce a representative crash and one to rerun a bounded campaign;
provide the bug list with disclosure status; and note which crashes are embargoed pending
vendor patches, offering redacted reproducers for those.
Output format
[Target badges] available / functional / reusable / results-reproduced
[Contents] <code / data / container / test vectors / logs>
[Turnkey level] one-command / scripted / descriptive / weak
[Safe-handling] <gating and disclosure posture for dangerous artifacts>
[Withheld-and-justified] <what and why, with substitute offered>
[Fixes before submission] <ordered list>
1---2name: ccs-artifact-evaluation3description: Use when packaging ACM CCS artifacts for the artifact-evaluation committee and the ACM badges — Artifacts Available, Artifacts Evaluated Functional, Artifacts Evaluated Reusable, and Results Reproduced — covering what security evaluators inspect, how to make attacks and defenses turnkey, and how to justify withheld artifacts.4---56# CCS Artifact Evaluation78Use this for artifact packaging around CCS. CCS runs an optional artifact-evaluation process9after acceptance; passing artifacts earn ACM badges that appear on the paper's first page.10Reopen the current call for artifacts for the exact badge set, submission form, and deadlines.1112## The ACM badge ladder1314| Badge | What it certifies | What the committee does |15|---|---|---|16| Artifacts Available | The artifact is publicly archived with a stable identifier | Confirms the artifact is retrievable and permanent |17| Artifacts Evaluated - Functional | The artifact runs and does what the paper says | Executes it against the documented steps |18| Artifacts Evaluated - Reusable | It exceeds functional quality and others can reuse it | Judges structure, documentation, and reusability |19| Results Reproduced | The main paper results are independently reproduced | Reruns experiments and checks they support the claims |2021Available is about archival permanence; Functional and Reusable are about quality; Results22Reproduced is the highest bar and requires the committee to regenerate your headline numbers.2324## Artifact plan2526- Decide which claims the artifact must support: the exploit, the defense overhead, the27 measurement pipeline, or the protocol implementation.28- Make the security claim turnkey: one documented command per headline result, with expected29 output, runtime, and the hardware assumed.30- Anonymize nothing at this stage — artifact evaluation is post-acceptance and single-blind or31 open — but do pin versions, dependencies, and a container so it runs on a clean machine.32- For dangerous artifacts (working exploits, malware, live attack tooling), gate access,33 document safe-handling, and follow the responsible-disclosure posture from the paper.34- Justify anything withheld: licensing, disclosure embargo, subject safety, or premature-release35 risk, and offer partial, synthetic, or redacted substitutes that still let evaluators check36 the method.3738## What security evaluators open first3940| Claim type | First artifact inspected | Common failure caught |41|---|---|---|42| Exploit against real software | The PoC and its target build instructions | Target version unspecified; PoC fails to build |43| Defense with overhead numbers | The benchmark runner and instrumented build | Overhead cannot be reproduced; workload undocumented |44| Measurement study | The collection and classification scripts | Classification rule differs from the paper's prose |45| Cryptographic implementation | Test vectors and the reference build | No test vectors; outputs cannot be checked |4647## Worked vignette: packaging a fuzzing-tool artifact4849A hypothetical accepted paper presents a new fuzzer that found N bugs. For Reusable and50Results Reproduced: ship a container pinning the fuzzer, the target corpus, and the seed set;51document one command to reproduce a representative crash and one to rerun a bounded campaign;52provide the bug list with disclosure status; and note which crashes are embargoed pending53vendor patches, offering redacted reproducers for those.5455## Output format5657```text58[Target badges] available / functional / reusable / results-reproduced59[Contents] <code / data / container / test vectors / logs>60[Turnkey level] one-command / scripted / descriptive / weak61[Safe-handling] <gating and disclosure posture for dangerous artifacts>62[Withheld-and-justified] <what and why, with substitute offered>63[Fixes before submission] <ordered list>64```
brycewang-stanford