Peer review of research software
Software peer review is the publication pathway where the SOFTWARE is
the reviewed artifact: JOSS (any language), rOpenSci (R) and
pyOpenSci (Python) run open, constructive reviews against public
checklists, and CODECHECK issues certificates that a paper's
computations were independently executed. For a maintainer, review
readiness is a concrete bar to build toward; passing it earns a
citable publication (JOSS papers get DOIs) and a community quality
mark. This is review of the whole package - distinct from PR-level
code review (rseng-version-control-review).
Pre-submission: self-review against the real checklist
Run the target venue's own checklist against the repository and fix
failures BEFORE submitting - reviewers check exactly these:
- License: OSI-approved LICENSE file at the root (rseng-licensing).
- Documentation entry points: installation that works from a clean
environment, usage examples that run, API docs for the public
surface (rseng-documentation).
- Statement of need: who this is for and what gap it fills - in the
README and (for JOSS) the paper; write it for the target
researcher, not the maintainer.
- Tests that run in CI, covering the core claims of the software
(rseng-testing, rseng-ci-cd); reviewers will run them.
- Community files: contributing guidelines and code of conduct
(rseng-community-governance).
- Citation metadata and archiving: CITATION.cff, and a Zenodo (or
equivalent) archive with DOI at acceptance
(rseng-citation-metadata, rseng-publishing-releasing).
- AI-usage disclosure: JOSS now asks about AI use in the submission -
the project's aidecl.yaml (rseng-ai-declaration) is exactly the
honest record to answer from.
Scope check before effort: each venue defines in-scope package types
and substantiality; read the venue's scope page first and say
honestly if the project is not there yet.
The JOSS paper: the user writes it
Short by design (750-1750 words): summary for
non-specialists, statement of need, rough state of the field
(neighboring tools and how this differs - rseng-discovery and
rseng-software-reuse habits help here), acknowledgements, references
with DOIs. It reviews the software; do not pad it into a methods
paper. The agent's role is support only: outline against the
venue's template, gather the material, check the draft against the
criteria and verify its citations - never produce a
submission-ready manuscript; authorship and every signed claim stay
with the user (rseng-science-communication states the same boundary;
JOSS itself asks about AI involvement - answer from aidecl.yaml).
Responding to reviews
Reviews are public issue threads. Respond to every point (fix,
discuss, or explain why not), push commits as you go, and summarize
changes when done. Tone: reviewers are volunteers improving your
software - thank them, and disagree with reasons, not defensiveness.
Reviewing and codechecking
When the user is the reviewer:
- Work through the venue checklist honestly - install from scratch
in a clean environment (rseng-reproducible-environments), run the
tests, run the examples; "it probably works" is not a review.
- File findings as actionable issues, most important first;
distinguish must-fix (checklist failures) from suggestions.
- CODECHECK mode: execute the paper's workflow, record what
reproduced (with outputs), what did not, and produce the
certificate-style summary of exactly what was checked.
- Constructive is the norm in this ecosystem: the goal is
acceptance-after-improvement, not gatekeeping.
Working with this skill
This skill is source-independent: its authority is the venues' own
review criteria and guides linked below.
Learn more (verified):
Related skills
Check whether any of these applies before moving on:
- rseng-ai-declaration - JOSS asks about AI use
- rseng-citation-metadata - CITATION.cff and DOI at acceptance
- rseng-discovery - state-of-the-field section material
- rseng-reproducible-environments - clean-room installs for reviewing
- rseng-software-publishing - JOSS within the channel mix
- rseng-version-control-review - PR-level review is distinct
1---2name: rseng-software-peer-review3description: Covers community peer review of research software: preparing a package for JOSS, pyOpenSci or rOpenSci submission, self-checking against their review criteria before submitting, writing the paper or statement of need, responding to reviews, and acting as a reviewer or CODECHECK-style codechecker who executes the artifact. Use when the user mentions JOSS, pyOpenSci, rOpenSci or CODECHECK, wants to submit software for peer review or publication, asks whether their package is review-ready, or is reviewing someone else's research software for one of these venues. (PR-level code review is rseng-version-control-review; overall publication channel strategy is rseng-software-publishing.)4license: CC-BY-4.05---67# Peer review of research software89Software peer review is the publication pathway where the SOFTWARE is10the reviewed artifact: JOSS (any language), rOpenSci (R) and11pyOpenSci (Python) run open, constructive reviews against public12checklists, and CODECHECK issues certificates that a paper's13computations were independently executed. For a maintainer, review14readiness is a concrete bar to build toward; passing it earns a15citable publication (JOSS papers get DOIs) and a community quality16mark. This is review of the whole package - distinct from PR-level17code review (rseng-version-control-review).1819## Pre-submission: self-review against the real checklist2021Run the target venue's own checklist against the repository and fix22failures BEFORE submitting - reviewers check exactly these:2324- License: OSI-approved LICENSE file at the root (rseng-licensing).25- Documentation entry points: installation that works from a clean26 environment, usage examples that run, API docs for the public27 surface (rseng-documentation).28- Statement of need: who this is for and what gap it fills - in the29 README and (for JOSS) the paper; write it for the target30 researcher, not the maintainer.31- Tests that run in CI, covering the core claims of the software32 (rseng-testing, rseng-ci-cd); reviewers will run them.33- Community files: contributing guidelines and code of conduct34 (rseng-community-governance).35- Citation metadata and archiving: CITATION.cff, and a Zenodo (or36 equivalent) archive with DOI at acceptance37 (rseng-citation-metadata, rseng-publishing-releasing).38- AI-usage disclosure: JOSS now asks about AI use in the submission -39 the project's aidecl.yaml (rseng-ai-declaration) is exactly the40 honest record to answer from.4142Scope check before effort: each venue defines in-scope package types43and substantiality; read the venue's scope page first and say44honestly if the project is not there yet.4546## The JOSS paper: the user writes it4748Short by design (750-1750 words): summary for49non-specialists, statement of need, rough state of the field50(neighboring tools and how this differs - rseng-discovery and51rseng-software-reuse habits help here), acknowledgements, references52with DOIs. It reviews the software; do not pad it into a methods53paper. The agent's role is support only: outline against the54venue's template, gather the material, check the draft against the55criteria and verify its citations - never produce a56submission-ready manuscript; authorship and every signed claim stay57with the user (rseng-science-communication states the same boundary;58JOSS itself asks about AI involvement - answer from aidecl.yaml).5960## Responding to reviews6162Reviews are public issue threads. Respond to every point (fix,63discuss, or explain why not), push commits as you go, and summarize64changes when done. Tone: reviewers are volunteers improving your65software - thank them, and disagree with reasons, not defensiveness.6667## Reviewing and codechecking6869When the user is the reviewer:7071- Work through the venue checklist honestly - install from scratch72 in a clean environment (rseng-reproducible-environments), run the73 tests, run the examples; "it probably works" is not a review.74- File findings as actionable issues, most important first;75 distinguish must-fix (checklist failures) from suggestions.76- CODECHECK mode: execute the paper's workflow, record what77 reproduced (with outputs), what did not, and produce the78 certificate-style summary of exactly what was checked.79- Constructive is the norm in this ecosystem: the goal is80 acceptance-after-improvement, not gatekeeping.8182## Working with this skill8384This skill is source-independent: its authority is the venues' own85review criteria and guides linked below.8687Learn more (verified):88 - https://joss.readthedocs.io/en/latest/review_checklist.html -89 JOSS review checklist90 - https://joss.theoj.org - Journal of Open Source Software91 - https://devguide.ropensci.org - rOpenSci packaging and review92 guide93 - https://www.pyopensci.org/software-peer-review/ - pyOpenSci peer94 review guide95 - https://codecheck.org.uk/ - CODECHECK independent execution96 certificates9798<!-- related-skills:begin -->99100## Related skills101102Check whether any of these applies before moving on:103104- rseng-ai-declaration - JOSS asks about AI use105- rseng-citation-metadata - CITATION.cff and DOI at acceptance106- rseng-discovery - state-of-the-field section material107- rseng-reproducible-environments - clean-room installs for reviewing108- rseng-software-publishing - JOSS within the channel mix109- rseng-version-control-review - PR-level review is distinct110111<!-- related-skills:end -->
fdiblen