Artifact pack
Template: templates/README_ARTIFACT.md (in this skill directory).
Goals (map to typical badges)
| badge (names vary by venue) |
what it needs |
| Available |
permanent public location (tagged repository or archive DOI) |
| Functional |
documentation, completeness, runs, can be extended by others |
| Reproduced |
an evaluator reruns the main experiments and the main claims hold within tolerance |
Layout
artifact/
README.md claims → commands → expected outputs → time/resources
LICENSE
env/ Dockerfile or environment pinning (requirements.txt with ==,
conda env.yml, or a Nix/Guix file); library commits pinned
baselines/MANIFEST.md third-party code: url + pinned commit/tag + build recipe
(fetch script, never vendored source trees)
scripts/
fetch_baselines.sh clones pinned commits
build.sh builds everything
smoke.sh < 10 minutes; checks every component runs and outputs verify
reproduce_table3.sh one script per paper table/figure
expected/ expected outputs (tables as csv, with tolerance notes)
results/ empty; scripts write here
EVIDENCE.md subset of the project ledger: paper number → script → log
Procedure
- Claims map. From
EVIDENCE.md, list every number/table/figure of the paper and
the script that regenerates it. Main claims first. Anything that cannot be
regenerated is marked "not reproduced by the artifact" with the reason.
- Pin everything: compiler version, library commits, Python packages with
==, estimator commit, random seeds. Record CPU/threads assumptions.
- Scripts: one command per table; print the environment (CPU, compiler, commit)
at the start; write logs to
results/; verify correctness (decryption/cleartext
reference) and fail loudly on mismatch.
- Expected outputs + tolerance: absolute times vary by machine; state which
quantities should match exactly (operation counts, parameters, security
estimates, correctness) and which only in ratio (speedups, with an expected band).
Explain that ratios come from interleaved runs in one session.
- Resource statement: RAM, disk, cores, expected wall time per script; offer a
reduced configuration for small machines.
- Smoke test under 10 minutes; evaluators often start there.
- Anonymous hosting during review: use an anonymising mirror service or a
zip in the supplementary material. Run
anonymize-check on the artifact
directory and the zip (.git excluded, no usernames/hosts in logs, README without
"our previous paper"). Refresh the mirror after the last push and check each file
URL returns 200. Keep the archive under the venue's supplementary size cap.
- After acceptance: de-anonymise, tag the release, archive (DOI), update the
paper's artifact URL, submit to the artifact track within its window.
Common failure modes
- Scripts depend on absolute paths or a local cache directory; use paths relative
to the artifact root and environment variables with defaults.
- Stale build directories used for measurements; always build from clean in scripts.
- Baseline built with a different, slower configuration than in the paper; document
flags in
baselines/MANIFEST.md.
- Numbers in the paper from a noisy session that the artifact cannot reproduce;
retract or re-measure before release.
- Hidden dependence on a proprietary tool (e.g. a commercial MILP solver or computer algebra system);
provide a free fallback or state it clearly.
1---2name: artifact-pack3description: Use when preparing supplementary code/data for review (anonymous) or for an artifact-evaluation track after acceptance. Produces a reproducible artifact with README, one-command scripts, pinned dependencies, expected outputs with tolerances, a claims-to-commands map, and anonymous hosting during review.4metadata:5 source: https://skillmd.com/skills/jiayaoqijia/artifact-pack6---78# Artifact pack910Template: `templates/README_ARTIFACT.md` (in this skill directory).1112## Goals (map to typical badges)1314| badge (names vary by venue) | what it needs |15|---|---|16| Available | permanent public location (tagged repository or archive DOI) |17| Functional | documentation, completeness, runs, can be extended by others |18| Reproduced | an evaluator reruns the main experiments and the main claims hold within tolerance |1920## Layout2122```23artifact/24 README.md claims → commands → expected outputs → time/resources25 LICENSE26 env/ Dockerfile or environment pinning (requirements.txt with ==,27 conda env.yml, or a Nix/Guix file); library commits pinned28 baselines/MANIFEST.md third-party code: url + pinned commit/tag + build recipe29 (fetch script, never vendored source trees)30 scripts/31 fetch_baselines.sh clones pinned commits32 build.sh builds everything33 smoke.sh < 10 minutes; checks every component runs and outputs verify34 reproduce_table3.sh one script per paper table/figure35 expected/ expected outputs (tables as csv, with tolerance notes)36 results/ empty; scripts write here37 EVIDENCE.md subset of the project ledger: paper number → script → log38```3940## Procedure41421. **Claims map.** From `EVIDENCE.md`, list every number/table/figure of the paper and43 the script that regenerates it. Main claims first. Anything that cannot be44 regenerated is marked "not reproduced by the artifact" with the reason.452. **Pin everything**: compiler version, library commits, Python packages with46 `==`, estimator commit, random seeds. Record CPU/threads assumptions.473. **Scripts**: one command per table; print the environment (CPU, compiler, commit)48 at the start; write logs to `results/`; verify correctness (decryption/cleartext49 reference) and fail loudly on mismatch.504. **Expected outputs + tolerance**: absolute times vary by machine; state which51 quantities should match exactly (operation counts, parameters, security52 estimates, correctness) and which only in ratio (speedups, with an expected band).53 Explain that ratios come from interleaved runs in one session.545. **Resource statement**: RAM, disk, cores, expected wall time per script; offer a55 reduced configuration for small machines.566. **Smoke test** under 10 minutes; evaluators often start there.577. **Anonymous hosting during review**: use an anonymising mirror service or a58 zip in the supplementary material. Run `anonymize-check` on the artifact59 directory and the zip (`.git` excluded, no usernames/hosts in logs, README without60 "our previous paper"). Refresh the mirror after the last push and check each file61 URL returns 200. Keep the archive under the venue's supplementary size cap.628. **After acceptance**: de-anonymise, tag the release, archive (DOI), update the63 paper's artifact URL, submit to the artifact track within its window.6465## Common failure modes6667- Scripts depend on absolute paths or a local cache directory; use paths relative68 to the artifact root and environment variables with defaults.69- Stale build directories used for measurements; always build from clean in scripts.70- Baseline built with a different, slower configuration than in the paper; document71 flags in `baselines/MANIFEST.md`.72- Numbers in the paper from a noisy session that the artifact cannot reproduce;73 retract or re-measure before release.74- Hidden dependence on a proprietary tool (e.g. a commercial MILP solver or computer algebra system);75 provide a free fallback or state it clearly.