thedixitjain EACL Artifact Evaluation
Use this to turn a paper's evidence into artifacts that survive review and become a public release. EACL runs through ACL Rolling Review, so the artifact lives two lives: an anonymized supplement attached at ARR submission, and a public release after commitment acceptance. Both are audited against the Responsible NLP checklist. Reopen the current checklist before packaging.
The two lives of an EACL artifact
| Stage | Form | Must be | Owner |
|---|---|---|---|
| ARR submission | Anonymized .zip/.tgz supplement |
Fully de-identified, self-contained | Authors |
| Commitment acceptance | Public repo + Anthology link | Licensed, versioned, reproducible | Authors |
Do not conflate them: the review supplement must contain no author-identifying strings, while the public release must contain exactly the identifying and licensing information the supplement omitted.
What belongs in an EACL artifact
- Code to reproduce the headline tables, with a top-level entry point.
- Data: the dataset or a loader plus a documented path to it; if redistribution is restricted, document access precisely rather than implying release.
- Prompts and decoding settings verbatim for any LLM-based result — these are part of the method, not an afterthought.
- Model outputs retained so scores can be re-computed without re-running expensive models.
- Annotation materials: guidelines, interface, pay information, and inter-annotator agreement.
Anonymized-supplement checklist
[ ] No author names in paths, file headers, LICENSE, or notebook metadata
[ ] Git history stripped or repo re-initialized
[ ] No personal hosting URLs (Drive/Dropbox) that identify authors
[ ] Prompts + decoding params included verbatim
[ ] Model outputs included for re-scoring
[ ] A README that reproduces at least one reported table
[ ] Smoke-checked (see resources/code/README.md)
Run the shared smoke checker before upload:
python3 ../../../shared-resources/ml-conference-methods/code/check_repro_package.py /path/to/anonymous-supplement
Licensing and documentation for the public release
- Choose a license appropriate to code (e.g. permissive) and data (respecting upstream dataset terms); the paper text should state it.
- Document intended use and known limitations of any released dataset — required by the checklist and expected by the European community's data-governance norms.
- Version the release with a tag that matches the camera-ready, so the Anthology PDF and the repo cannot drift.
Multilingual and lower-resource specifics
- If the artifact covers lower-resourced languages, document provenance and speaker/annotator context carefully; thin documentation of a low-resource dataset is a common EACL reviewer concern.
- Keep language codes and scripts explicit (ISO codes, script variants) so the artifact is usable by others working on those languages.
Output format
[Artifact stage] Anonymized supplement / Public release
[Contents] <code/data/prompts/outputs/annotation coverage>
[Anonymization] <pass/fail with specific leaks>
[Reproduces] <which reported table the README regenerates>
[Licensing + docs] <license, dataset terms, intended-use note>
[Gaps] <what a reviewer could still not reproduce>
Source: brycewang-stanford/Awesome-Journal-Skills → EACL-Skills/skills/eacl-artifact-evaluation/SKILL.md