Rseng Workflows

Computational workflows

Use this skill when a user is automating a multi-step, often multi-tool data analysis or data-handling pipeline and wants it to be reproducible, scalable, and shareable. A computational workflow is a special kind of software that describes what should be done - a series of actions and the data flowing between them - and hands the how (scheduling, resource management, provenance tracking) to a workflow management system. Steer toward this approach when automation would boost efficiency, when analyses must be re-run across many datasets, or when infrastructures need integrating without reworking existing processes.

When to adopt a workflow manager

Treat these as triggers to move a user off ad-hoc glue and onto a WMS:

• The pipeline chains multiple tools/scripts and is run repeatedly, on new data, or by other people.
• Steps must run on an HPC cluster or cloud, or need parallelism the laptop cannot provide.
• Reproducibility, provenance, or transparent documentation is a requirement (publication, review, regulatory, or "future self" reruns).
• A hand-written Bash script or a notebook full of ordered cells has become hard to maintain, extend, or share - the classic failure mode of the manual "set of instructions" approach.

Recognise the spectrum, and place the user on it rather than forcing a jump:

• General-purpose scripting (Bash) - lowest ceremony, hardest to maintain and share at scale.
• Electronic research notebooks (Jupyter, RStudio, Apache Zeppelin) - ordered stages, good for exploration, still manual plumbing.
• Dedicated WMS - highest automation, reproducibility, and scalability because it separates inputs from data flow and abstracts the run mechanics behind a high-level definition language.

In practice these combine: a WMS can chain simple scripts, launched from a notebook. Do not oversell a WMS for a one-off, single-step script.

Choose a workflow system

There is no universal best engine; the choice is driven by community, facility, and environment. Work through these factors with the user before naming a tool:

• Domain and community - scientific communities standardise around particular systems (e.g. REANA for particle physics; Galaxy, Snakemake, and Nextflow with deep roots in the life sciences). Matching the community buys shared workflows, documentation, and support.
• Facility and computing infrastructure - the target cluster or cloud may only support or optimise certain engines; this matters most when scaling.
• Availability of ready-made workflows, docs, and user support in that domain.

Common systems and their definition languages (pair the engine with its language when advising):

• Nextflow - Nextflow DSL.
• Snakemake - Snakefile.
• Galaxy - browser-based, workflow repository and ToolShed.
• Apache Airflow - Airflow DAGs (Python).
• Parsl - Python-native parallel workflows.
• Cross-engine definition standards worth knowing: Common Workflow Language (CWL) and Workflow Description Language (WDL), which several registries and engines accept.

For a rigorous comparison, point users to "A Terminology for Scientific Workflow Systems", which classifies 23 WMSs by composition, orchestration, data management, and metadata capture, and to the domain review papers for life-sciences engine choice.

Discover before you build

Reusing an existing workflow beats writing one from scratch - it saves time, avoids duplication, and inherits peer-reviewed, validated building blocks, which raises trust in results and aids reproducibility. Advise users to search registries first, then register their own so others can reuse them.

Point to the right registry for the context:

• WorkflowHub - cross-discipline FAIR workflow registry, strong life-science uptake; auto-registers nf-core workflows.
• nf-core - curated, community-reviewed Nextflow pipelines.
• Dockstore - registry supporting CWL, WDL, Nextflow, and Galaxy.
• Galaxy ToolShed / Galaxy workflow repository - Galaxy-compatible tools and workflows.
• REANA - reusable, reproducible analyses in particle physics and related fields.
• GESIS Methods Hub - emerging hub for social-science and digital behavioural workflows.
• GitHub / GitLab - general-purpose sharing alongside code and docs; note that GitHub search treats some workflow languages (e.g. CWL) as a searchable Language.

Describe workflows with metadata

A workflow definition tells a WMS how to execute; workflow metadata describes the workflow so humans and machines can understand, manage, and discover it - purpose, inputs/outputs, dependencies, authorship, and provenance. Both are needed; do not conflate them.

• Recommend Workflow RO-Crate for packaging: it bundles the workflow definition (CWL, Nextflow DSL, Snakefile, etc.) with contextual metadata in a FAIR-compliant crate. It is the standard used by WorkflowHub and the LifeMonitor service.
• It builds on the RO-Crate standard and extends the schemas.science ComputationalWorkflow profile (from Bioschemas), using FormalParameter to describe inputs/outputs.
• Payoff to state: more discoverable, portable and reproducible workflows, especially when published alongside datasets or in a registry.

Make workflows FAIR

Workflows are digital objects to be shared, discovered, and reused, so apply the FAIR principles - both the general FAIR-for-research-software recommendations and the dedicated FAIR recommendations for workflows. Concretely, when reviewing or authoring a workflow, check that it is:

• Findable - registered with rich metadata and a persistent identifier (e.g. deposited in WorkflowHub with a DOI).
• Accessible - retrievable via its identifier, with the definition and metadata openly available.
• Interoperable - written in a recognised definition language and packaged with standard metadata (Workflow RO-Crate) so other tools can consume it.
• Reusable - clearly licensed, documented, versioned, and modular, with the computational environment pinned (see reproducible software environments, a closely related concern).

Point users at the Workflows Community Initiative FAIR Workflows Working Group (WCI-FW) as the authoritative, cross-domain community effort on applying FAIR data and software principles to workflows.

Benefits to name when advocating

When a user is undecided, ground the pitch in concrete gains: reproducibility (each step, tool, parameter, and environment is formalised and rerunnable); automation and scaling (run unattended across datasets, distribute over HPC/cloud); transparency and provenance (living documentation of the method); and modularity, reuse, collaboration, and sharing (swap components, run the same process across systems and teams).

Working with this skill

The generated references.md beside this file lists the source material and pointers:

• references.md - verified Learn more pointers

Learn more (verified):

• https://snakemake.readthedocs.io - Snakemake workflow management system
• https://docs.seqera.io/nextflow/ - Nextflow documentation
• https://www.commonwl.org - Common Workflow Language standard
• https://workflowhub.eu - registry for computational workflows
• https://doi.org/10.1038/s41597-025-04451-9 - applying FAIR principles to computational workflows

Related skills

Check whether any of these applies before moving on:

• rseng-big-data-processing - scaling pipelines across many datasets
• rseng-discovery - finding reusable workflows in registries
• rseng-fair-software - FAIR principles applied to workflows
• rseng-hpc-computing - running stages on clusters
• rseng-provenance - engine logs are provenance capture
• rseng-reproducible-environments - per-step pinned environments