# Corl Artifact Evaluation > Use when packaging the artifacts of a CoRL paper — code, training configs, demonstration datasets, policy checkpoints, simulation environments, and benchmark definitions — as anonymous review-time evidence and as durable public releases after acceptance, in a venue with no formal artifact-badging track. - Skill: `thedixitjain/corl-artifact-evaluation` (Agent Skill) - Install (CLI): `npx skillmds add thedixitjain/corl-artifact-evaluation` - Raw SKILL.md: https://api.skillmd.com/api/skills/thedixitjain/corl-artifact-evaluation/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: thedixitjain (https://skillmd.com/u/thedixitjain) - Updated: 2026-09-08 - Page: https://skillmd.com/skills/thedixitjain/corl-artifact-evaluation --- # CoRL Artifact Evaluation CoRL has no separate artifact-evaluation track or badge system in the 2026 materials verified for this pack (a formal AE track remains 待核实 each cycle). Artifacts are instead judged twice, informally: by reviewers deciding whether to trust your tables, and by the community deciding — for years afterward — whether your method becomes a baseline. Package for both audiences from the start. ## Artifact inventory for a robot-learning paper | Artifact | Review-time form (anonymous) | Public form (post-acceptance) | |---|---|---| | Training code + configs | Scrubbed ZIP in the supplementary | GitHub repo, tagged release matching the paper | | Evaluation harness | Same ZIP; scripts + fixed init-state lists | Same repo; the part reused most by others | | Demonstration data | Small sample in ZIP; full set described | Archive with DOI + datasheet, license stated | | Policy checkpoints | Optional if small; else described | Hosted weights keyed to each results table | | Sim environments / tasks | Env definitions + versions in ZIP | Repo or upstream PR to the benchmark suite | | Hardware recipe | Appendix spec (rig, control interface) | Project page: BOM-level detail, photos | | Overview video | Supplementary upload (≤ 250 MB, 2026 cap) | Project page / video host — PMLR takes no video | ## The reuse test Design the release around one question: *can a stranger reproduce your evaluation without emailing you?* Concretely, a competent robot-learning grad student should be able to (1) install, (2) run evaluation with a released checkpoint, and (3) regenerate one paper table, in an afternoon, in simulation. Training reproduction and hardware reproduction are aspirational tiers above this floor — label the tiers honestly rather than implying all three. ```text Release tiers — declare one per artifact in the README: T1 evaluate: released checkpoint + eval script reproduce Table N in sim T2 retrain: configs + data (or data recipe) reproduce the training run within the reported seed spread T3 re-embody: rig documentation sufficient to attempt hardware replication ``` ## Review-time packaging (anonymous) - One ZIP, one top-level README, a table mapping `paper table → command`. - Determinize what you can: fixed evaluation seeds, fixed init-state lists, pinned dependency versions (`corl-reproducibility` owns the manifest format). - Anonymize mechanically, then by eyeball: repo history stripped, paths cleaned, no W&B/HF org names, no grant numbers in license headers, and remember the URL rule — an anonymized page on a lab-named domain still leaks. - Keep it runnable without a robot: reviewers do not have your hardware, so the artifact's demonstrable slice is sim evaluation plus logs/video of the hardware runs. Ship episode-level logs (CSV) for every hardware table. ## Post-acceptance: durable release - **Timing.** The 2026 camera-ready deadline (October 12) is when links get frozen into the PMLR record — stand up the public repo, dataset archive, and project page *before* finalizing the PDF so the printed URLs are real. - **Durability ladder.** Lab web servers die with funding cycles: put datasets and static artifacts in DOI-granting archives, code in a tagged repository release, and treat the project page as a pointer hub rather than the sole home. - **Checkpoint provenance.** Key each released checkpoint to its table and seed ("`ckpt_t3_s2` = Table 3, seed 2"); publishing only a best-seed checkpoint while the paper reports seed means invites a mismatch report you'll answer publicly — CoRL reviews of accepted papers are public, and so is the follow-up scrutiny. - **License deliberately.** Code (permissive vs copyleft), data (usage terms, consent constraints for human video), and weights (increasingly their own license class) are three separate decisions; "no license" means "nobody may legally build on this." ## Benchmarks and datasets as first-class artifacts If the artifact *is* the contribution (a task suite, a large demo corpus), the bar rises from reuse to stewardship: - Version the benchmark explicitly (v1.0 at camera-ready) and changelog any post-publication fix; silent edits corrupt every downstream comparison. - Publish the evaluation protocol as executable code, not prose — episode counts, init distributions, success criteria — so future papers cite numbers produced by *your* harness rather than reimplementations. - Provide a datasheet: collection method, operator demographics if teleop, filtering, known biases, consent/licensing status of any human footage. ## Failure patterns that surface after publication Because CoRL reviews of accepted papers are public and the community reuses artifacts aggressively, release defects surface fast and visibly: - **The vanished environment**: the eval harness imports a benchmark fork whose branch was deleted; vendor the environment code or pin an archived copy. - **The config drift**: the repo's default config differs from the paper's runs; ship the exact configs used, named per table, and make defaults match. - **The GPU-only afternoon**: an evaluation that silently requires 8×A100 to run at all; document minimum hardware and offer a reduced smoke-test target. - **The consent surprise**: teleop or human-video data released without the consent scope to permit it; resolve data-rights questions before the paper promises release, not after. ## Release checklist ```text [ ] Tier (T1/T2/T3) declared per artifact; T1 actually tested by a teammate on a clean machine [ ] README: install steps, table→command map, expected tolerances [ ] Checkpoints keyed to tables and seeds; selection rule stated [ ] Dataset archived with DOI, datasheet, and license [ ] Hardware logs + uncut eval video published for hardware tables [ ] Licenses chosen for code, data, and weights separately [ ] All URLs live before camera-ready freeze (2026: Oct 12 AoE) [ ] Anonymous variant retired only after decisions are out ``` Re-check the live cycle for any new artifact policy, badge program, or required availability statement at https://www.corl.org/contributions/instruction-for-authors — venue policy here is young and moves quickly, like the field itself. --- **Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `CoRL-Skills/skills/corl-artifact-evaluation/SKILL.md`