How do I install the Eacl Reproducibility skill?

Run npx skillmds@latest add brycewang-stanford/eacl-reproducibility in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Eacl Reproducibility safe to use?

This skill has not completed SkillMD's automated safety review yet. Capability flags: docs only. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Eacl Reproducibility?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Eacl Reproducibility free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Eacl Reproducibility?

brycewang-stanford (@brycewang-stanford) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Eacl Reproducibility by brycewang-stanford

Name: Eacl Reproducibility
Rating: 4.5 (4 reviews)
Author: brycewang-stanford

Eacl Reproducibility

Use when running the Responsible NLP checklist as a claims audit for an EACL paper, covering hyperparameter and compute disclosure, verbatim prompt and decoding reporting, data-contamination stance, variance and significance reporting, multilingual coverage claims, and consistency between the checklist answers and what the paper actually contains.

brycewang-stanford Updated Sep 8, 2026 1k repo stars

File contents

EACL Reproducibility

Use this to audit an EACL paper against the Responsible NLP checklist, which ARR files at submission and which the action editor and reviewers read alongside the PDF. The checklist is not paperwork: misleading answers are desk-rejection grounds, and inconsistencies between the checklist and the paper are what careful EACL reviewers hunt for. Reopen the current checklist at aclrollingreview.org/responsibleNLPresearch before auditing.

Treat the checklist as a claims audit

Every "yes" in the checklist implies a location in the paper. Walk the paper claim by claim and confirm each is backed:

Claim type	Must disclose	Common EACL failure
Model results	Hyperparameters, tuning, model size	"Default settings" with no numbers
Compute	Hardware, run time, total budget	Silent on cost of large runs
LLM prompts	Verbatim prompts + decoding params	Paraphrased or omitted prompts
Data	Source, license, splits, preprocessing	Undocumented or "on request" data
Metrics	Variance over seeds / significance test	Single-run deltas reported as wins
Multilingual	Languages, resource levels, per-language results	Aggregate hides where it fails

Contamination stance (LLM era)

State explicitly whether evaluation data could have leaked into training — for closed LLMs this is often unknowable, and the honest move is to say so and bound the risk rather than claim clean evaluation. The "Leak, Cheat, Repeat" exemplar in ../../resources/exemplars/library.md is the reference discipline.
Where feasible, run an overlap or decontamination check and report it.

Variance and significance floor

Reporting rule of thumb:
  - >= 3-5 seeds for any headline comparison
  - report mean +/- CI or std, not a single run
  - a significance test when two systems are "close"
  - never claim a win on an unreplicated single-run delta

Multilingual coverage honesty

If the paper claims a cross-lingual or multilingual result, the checklist audit must confirm the languages are named, the resource levels are stated, and per-language results exist somewhere — an aggregate average is not evidence for every language.
For lower-resourced languages, confirm dataset provenance and annotation context are documented; this is a recurring EACL reviewer expectation.

Checklist-to-paper consistency sweep

[ ] Every checklist "yes" maps to a section/appendix number
[ ] Hyperparameters + search space stated
[ ] Compute budget stated for expensive runs
[ ] Prompts + decoding params verbatim (if LLMs used)
[ ] Data source, license, splits, preprocessing documented
[ ] Variance/significance reported for headline claims
[ ] Contamination risk addressed honestly
[ ] Per-language results present for multilingual claims
[ ] AI-assistance use disclosed truthfully

Output format

[Reproducibility risk] Low / Medium / High
[Checklist-paper mismatches] <specific "yes" answers not backed in text>
[Disclosure gaps] <hyperparameters/compute/prompts/data>
[Evidence floor] <seeds/variance/significance findings>
[Contamination] <stance + any check run>
[Fix order] <what to add before the cycle deadline>

---name: eacl-reproducibilitydescription: Use when running the Responsible NLP checklist as a claims audit for an EACL paper, covering hyperparameter and compute disclosure, verbatim prompt and decoding reporting, data-contamination stance, variance and significance reporting, multilingual coverage claims, and consistency between the checklist answers and what the paper actually contains.---# EACL ReproducibilityUse this to audit an EACL paper against the **Responsible NLP checklist**, which ARR files atsubmission and which the action editor and reviewers read alongside the PDF. The checklist isnot paperwork: **misleading answers are desk-rejection grounds**, and inconsistencies betweenthe checklist and the paper are what careful EACL reviewers hunt for. Reopen the currentchecklist at `aclrollingreview.org/responsibleNLPresearch` before auditing.## Treat the checklist as a claims auditEvery "yes" in the checklist implies a location in the paper. Walk the paper claim by claim andconfirm each is backed:| Claim type | Must disclose | Common EACL failure ||---|---|---|| Model results | Hyperparameters, tuning, model size | "Default settings" with no numbers || Compute | Hardware, run time, total budget | Silent on cost of large runs || LLM prompts | Verbatim prompts + decoding params | Paraphrased or omitted prompts || Data | Source, license, splits, preprocessing | Undocumented or "on request" data || Metrics | Variance over seeds / significance test | Single-run deltas reported as wins || Multilingual | Languages, resource levels, per-language results | Aggregate hides where it fails |## Contamination stance (LLM era)- State explicitly whether evaluation data could have leaked into training — for closed LLMs this is often unknowable, and the honest move is to **say so and bound the risk** rather than claim clean evaluation. The "Leak, Cheat, Repeat" exemplar in `../../resources/exemplars/library.md` is the reference discipline.- Where feasible, run an **overlap or decontamination check** and report it.## Variance and significance floor```textReporting rule of thumb: - >= 3-5 seeds for any headline comparison - report mean +/- CI or std, not a single run - a significance test when two systems are "close" - never claim a win on an unreplicated single-run delta```## Multilingual coverage honesty- If the paper claims a **cross-lingual** or **multilingual** result, the checklist audit must confirm the languages are named, the resource levels are stated, and **per-language results** exist somewhere — an aggregate average is not evidence for every language.- For lower-resourced languages, confirm dataset provenance and annotation context are documented; this is a recurring EACL reviewer expectation.## Checklist-to-paper consistency sweep```text[ ] Every checklist "yes" maps to a section/appendix number[ ] Hyperparameters + search space stated[ ] Compute budget stated for expensive runs[ ] Prompts + decoding params verbatim (if LLMs used)[ ] Data source, license, splits, preprocessing documented[ ] Variance/significance reported for headline claims[ ] Contamination risk addressed honestly[ ] Per-language results present for multilingual claims[ ] AI-assistance use disclosed truthfully```## Output format```text[Reproducibility risk] Low / Medium / High[Checklist-paper mismatches] <specific "yes" answers not backed in text>[Disclosure gaps] <hyperparameters/compute/prompts/data>[Evidence floor] <seeds/variance/significance findings>[Contamination] <stance + any check run>[Fix order] <what to add before the cycle deadline>```

Frequently asked questions