brycewang-stanford DAC Artifact Evaluation
Read the venue reality first. Unlike the software-engineering venues (FSE/ICSE/ISSTA) and the
computer-architecture venues (MICRO/ISCA/HPCA), DAC has historically not operated a standing,
badge-issuing artifact-evaluation track for research manuscripts; whether the current cycle adds
one is 待核实 (resources/official-source-map.md). So do not build for an ACM badge that DAC
does not award. Build instead for the two things that actually matter at DAC: making a skeptical
reviewer trust your QoR numbers and making the community able to reuse and cite your work.
What an EDA artifact is for at DAC
| Goal | What it buys you | How it is earned |
|---|---|---|
| Reviewer credibility | A QoR claim that reads as real, not cherry-picked | Standard benchmarks, disclosed flow, a runnable path |
| Community reuse | Citations and follow-on work built on your tool | Clean open-source release, good docs, a license |
| Contest/leaderboard standing | Recognition on ISPD/TAU/CAD-contest tasks | A tool that runs on the contest harness |
There is no badge to chase, so the design target is a package a stranger can run and a reviewer would believe — not a checklist an evaluator scores.
Packaging plan for an EDA artifact
[Flow] pin the toolchain: OpenROAD / ABC / Yosys / KLayout versions, or a Docker image;
avoid "install these commercial tools yourself" as the only path
[Benchmarks] ship or clearly reference the exact benchmark release (ISPD/EPFL/ISCAS/ITC/TAU/
CircuitNet) with its version
[PDK/library] include the open PDK/library used (Nangate/ASAP7) or document the NDA-gated one
[README] what it is; how to install; how to run one small demo in minutes; how to reproduce
each headline QoR number; expected runtime and outputs
[Mapping] an explicit table: paper claim -> script + benchmark -> expected QoR result
[Seeds] fixed seeds and a note on nondeterminism for stochastic/RL flows
[License] an OSI-approved license so others can build on it
[Archive] deposit a versioned release in a DOI-issuing archive (Zenodo/Software Heritage) after
acceptance for a stable citation
The clean-machine, bounded-time test
Even without formal evaluators, assume a reviewer or a future user gives your package a short, bounded try on a clean machine. Design the first ten minutes to succeed:
- A one-command small demo that runs a tiny circuit through your tool and prints a recognizable QoR number.
- No dependency on the authors' cluster, private PDK, or a live license server for the demo path.
- A clear separation between the fast demo and the slow full-benchmark reproduction (which may take hours on large designs — say so).
Anonymized review artifact vs public release
- At submission (double-blind): if you link the artifact for reviewers, anonymize it exactly like the paper — no author/lab names, no personal repos, no cluster paths, no commercial-flow fingerprints. Reviewers are not required to open it, so it can only strengthen a paper that already stands on six pages.
- After acceptance: replace anonymized placeholders with the public, licensed, DOI-archived release cited in the camera-ready; this is the version the community reuses.
Worked vignette: packaging a placement tool + benchmarks
A paper contributes an ML-guided placer evaluated on ISPD circuits. A credible, reusable package:
a Docker image with the placer and an OpenROAD flow pre-built; a run_demo.sh that places one small
ISPD circuit and reports wirelength/DRC in under a minute; a reproduce/ directory whose scripts
regenerate each per-benchmark table with fixed seeds; a claim-to-script-to-benchmark mapping in the
README; the exact ISPD release and an open PDK; the trained model checkpoint and the train/test
design split; and an MIT/Apache license. State plainly which numbers are turnkey and which need the
slow full-suite run on a large machine.
Calibration
- Do not assume a DAC artifact badge exists; verify the current cycle before promising one.
- If DAC or a co-located workshop does introduce an artifact/open-source track in a given year, re-read its specific rules — do not assume ACM SIGSOFT-style badge names transfer.
- The payoff is credibility and citations; budget the artifact accordingly, not as a graded deliverable.
Output format
[Artifact goal] reviewer credibility / community reuse / contest harness
[Contents] tool + flow versions / benchmarks + version / PDK / model + split / license
[Ten-minute test] does install + small demo succeed on a clean machine? yes/no
[Claim mapping] claim -> script -> benchmark -> expected QoR present? yes/no
[Anon vs public] anonymized review artifact / DOI-archived public release planned? yes/no
[Badge check] does this cycle actually run an artifact track? verify -> yes/no/待核实