Nextflow Profile Customization

nextflow-profile-customization

Summary

Customize Nextflow execution profiles to support containerized metabolomics workflows across diverse computational environments (local Docker, HPC with Singularity). This skill enables portable, reproducible LC-HRMS data processing by configuring container runtimes, resource directives, and tool-specific parameters for target execution contexts.

When to use

You need to execute a Nextflow metabolomics workflow (e.g., MS-DIAL–based LC-HRMS processing) on a new computational environment—such as an HPC cluster with Singularity support or a local macOS/Linux machine with Docker—and existing profiles do not match your resource constraints, container runtime, or job scheduler (e.g., Slurm queue allocation).

When NOT to use

• Input data is already in feature-table format (e.g., CSV abundance matrix); use this skill only when raw .mzML/.abf spectra must be processed through MS-DIAL.
• Target environment does not support containerization (no Docker or Singularity installation available); Nextflow4MS-DIAL requires container runtime for reproducibility.
• You are running on an unsupported or untested HPC system where Singularity configuration details are unknown and no reference profile (e.g., HiPerGator.config analogue) exists—profile customization may fail without HPC-specific documentation.

Inputs

• .mzML or .abf raw LC-MS metabolomics data files
• MS-DIAL configuration file (msdial_params.txt)
• MS-FLO configuration file (msflo_params.ini)
• MS1 and MS2 spectral library files (ms1_lib.txt, ms2_lib.msp)
• Nextflow project repository (nextflow.config, conf/*.config files)

Outputs

• .msdial output files (converted to .tsv for spreadsheet compatibility)
• execution_report.html (workflow runtime and resource usage summary)
• execution_timeline.html (per-process execution timeline)
• logs/execution.log (detailed execution metadata and process logs)
• Validated nextflow.config profile for target environment

How to apply

1. Identify the target execution context (e.g., HPC cluster with Slurm scheduler, local Docker-capable machine) and its resource limits (CPU, memory, queue names). 2. Review the project's nextflow.config and existing profiles (e.g., conf/base.config, conf/HiPerGator.config) to understand current container directives and resource allocation patterns. 3. Create or extend a profile in nextflow.config with: (a) container enablement (e.g., singularity.enabled = true or docker.enabled = true), (b) process directives specifying container image paths (e.g., process.container = 'singularity://...'), and (c) HPC-specific resource requests (cpus, memory, queue, time limits). 4. Use --max_cpus (not --cpus) to define per-process CPU limits relative to available cluster resources. 5. Test the profile on sample .mzML LC-MS data (or a functional_test profile) by running nextflow run main.nf -profile and inspect execution_report.html and logs for container initialization success, resource utilization, and process completion. 6. Validate that all output .msdial files are produced and that no resource-related errors (e.g., 'Process requirement exceed available CPUs') appear in error logs.

Related tools

• Nextflow (Workflow orchestration engine; manages process execution, parallelization, and container runtime selection across local and HPC environments) — https://www.nextflow.io/
• Singularity (Container runtime for HPC environments; executes MS-DIAL and MSFLO processes in isolated, reproducible container images on compute clusters) — https://www.sylabs.io/guides/3.0/user-guide/
• Docker (Container runtime for local execution; packages MS-DIAL and MSFLO tools with all dependencies for macOS and Linux workstations) — https://docs.docker.com/engine/installation/
• MS-DIAL (Peak detection and metabolite identification tool; processes .mzML LC-HRMS data according to user-defined parameters in msdial_params.txt) — https://github.com/Nextflow4Metabolomics/nextflow4ms-dial
• MSFLO (Post-processing and feature annotation tool; refines MS-DIAL output and generates final metabolite abundance tables) — https://github.com/Nextflow4Metabolomics/nextflow4ms-dial

Examples

nextflow run main.nf -profile singularity --max_cpus 20

Evaluation signals

• Container initialization succeeds: execution_report.html and logs confirm 'Singularity [version]' or 'Docker [version]' runtime active with no 'container not found' or 'pull failed' errors.
• Resource allocation matches request: execution_report.html shows actual CPU and memory usage ≤ specified limits; no Slurm errors like 'Process requirement exceed available CPUs' in error.txt.
• All processes complete: execution_timeline.html displays all pipeline steps (MS-DIAL, MSFLO) in green (completed); .msdial/.tsv output files exist in results/ directory with non-zero file size.
• Profile-specific parameters applied: logs/execution.log lists correct queue name, cpus, and memory values for the target profile; container image path matches nextflow.config specification.
• No file-name errors: output files lack special characters (except underscores); no 'file not found' or 'invalid filename' messages in error logs.

Limitations

• Special characters in input file names (except underscores) cause silent process skipping; users must rename raw .mzML/.abf files before execution.
• CPU limit parameter --max_cpus must be used instead of --cpus in HPC configuration; incorrect parameter name causes resource allocation failures with cryptic Slurm messages.
• Profile customization requires manual HPC environment documentation (queue names, max CPU/memory, job submission system); automated detection not supported.
• Singularity container images must be pre-built or pulled from a compatible registry; Docker-to-Singularity conversion requires additional tools and expertise outside Nextflow.
• HPC scheduler differences (Slurm vs. LSF vs. PBS) necessitate environment-specific profiles; a single profile may not port across institutions.

Evidence

• [other] Both Docker and Singularity (for high-performance computing) are supported: "Both Docker and Singularity (for high-performance computing) are supported"
• [readme] High-performance computing and Singularity configuration is defined in conf/HiPerGator.config.: "High-performance computing and Singularity configuration is defined in conf/HiPerGator.config."
• [other] Create a Singularity profile in the nextflow.config that specifies the container directive with Singularity syntax (e.g., singularity.enabled = true, process.container = 'singularity://...') and HPC resource directives (queue, cpus, memory).: "Create a Singularity profile in the nextflow.config that specifies the container directive with Singularity syntax (e.g., singularity.enabled = true, process.container = 'singularity://...') and HPC"
• [readme] Use --max_cpus, not --cpus, in the configuration file to define the CPUs available to each process.: "Use --max_cpus, not --cpus, in the configuration file to define the CPUs available to each process."
• [readme] To avoid unexpected errors, do not use special characters in file names. Underscores are safe to use.: "To avoid unexpected errors, do not use special characters in file names. Underscores are safe to use."
• [readme] nextflow run main.nf -profile docker for local execution and the singularity profile for high-performance computing environments: "nextflow run main.nf -profile docker for local execution and the singularity profile for high-performance computing environments"
• [readme] execution_report.html summarizes workflow runtime and computational resource usage.: "execution_report.html summarizes workflow runtime and computational resource usage."