batch-file-processing-orchestration
Summary
Orchestrate batch ingestion of multiple CDF (NetCDF) imaging files into a structured Matlab workspace, aggregating spectral data, m/z values, and spatial coordinates for downstream mass spectrometry analysis. This skill enables efficient parallel or sequential processing of large imaging datasets typical in developmental mass spectrometry imaging workflows.
When to use
When you have multiple CDF imaging files (e.g., from mass spectrometry imaging scans of biological samples) that need to be read into a single Matlab workspace with consistent structure and metadata (spectral intensity, m/z arrays, spatial coordinates). Typical trigger: a directory or file list containing 3+ CDF files destined for a linear or spatial imaging pipeline like DIMPLE.
When NOT to use
- Input files are already merged into a single CDF or HDF5 file — use direct read instead.
- Spectral data are in non-imaging formats (e.g., single MS profiles without spatial metadata) — use simpler file I/O.
- You require real-time streaming or on-disk processing of files too large to aggregate in memory — use chunked or memory-mapped I/O instead.
Inputs
- cell array of CDF file paths
- file list or directory containing CDF imaging files
- NetCDF/CDF file format spectra with embedded m/z and spatial metadata
Outputs
- aggregated Matlab workspace structure with all batch spectra
- cell array or struct array with fields: spectra (intensity matrices), m/z values, spatial coordinates
- Matlab .mat file suitable for downstream DIMPLE pipeline analysis
How to apply
Parse the input CDF file paths as a cell array or file list. Iterate through each CDF file using Matlab's NetCDF/CDF I/O functions (e.g., netcdf functions or dedicated CDF readers). For each file, extract spectral intensity arrays, m/z value vectors, and spatial coordinates (x, y position or linear position along the root/sample axis). Organize extracted data into a structured array or cell array with consistent fields (spectra, m/z, spatial metadata). Return the aggregated workspace structure containing all batch spectra in a format compatible with downstream pipeline analysis (e.g., DIMPLE filtering and segmentation). Validate that all spectra share the same m/z axis and that spatial metadata align with the sample geometry.
Related tools
- Matlab NetCDF/CDF I/O functions (Read spectral intensity arrays, m/z values, and spatial metadata from individual CDF files)
- batchcdfread function (Custom batch orchestration function developed in Dr. Zare's lab; wraps NetCDF reads and aggregates output into workspace structure) — github.com/dickinsonlab/DIMPLE-code
- DIMPLE pipeline (Downstream consumer of aggregated batch workspace; performs filtering, segmentation, and linear imaging analysis) — github.com/dickinsonlab/DIMPLE-code
Evaluation signals
- Output workspace contains one entry per input CDF file with no missing or truncated spectra.
- All spectra in the aggregated structure share the same m/z axis (vector length and values match across batch).
- Spatial coordinates are populated and consistent with sample geometry (e.g., linear positions 1 to N for N-scan root imaging).
- Output struct or cell array fields (spectra, m/z, spatial metadata) are present and non-empty for every batch member.
- Aggregated workspace can be directly loaded and processed by DIMPLE filtering/segmentation functions without schema errors.
Limitations
- Requires sufficient RAM to hold all spectra in memory; large imaging datasets may exceed available workspace.
- Assumes all CDF files follow a consistent internal structure and metadata naming convention (e.g., all contain 'intensity', 'm/z', 'x', 'y' or 'position' fields); heterogeneous or malformed CDF files may cause read failures.
- No built-in error recovery or partial batch completion; a single corrupted CDF file halts the entire batch unless wrapped in error handling.
- Performance scales linearly with file count; no parallelization mentioned in the source material.
Evidence
- [readme] batchcdfread function developed by Yifan Meng in Dr. Richard N. Zare's lab at Stanford University: "batchcdfread function developed by Yifan Meng in Dr. Richard N. Zare's lab at Stanford University"
- [other] Iterate through each CDF file and read the spectral data using Matlab's netCDF/CDF I/O functions. 3. Extract spectral intensity arrays, m/z values, and spatial coordinates: "Iterate through each CDF file and read the spectral data using Matlab's netCDF/CDF I/O functions. 3. Extract spectral intensity arrays, m/z values, and spatial coordinates (x, y, or linear position)"
- [other] Organize extracted data into a structured array or cell array with fields for spectra, m/z values, and spatial metadata. 5. Return the aggregated workspace structure containing all batch spectra in a format compatible with downstream DIMPLE pipeline analysis.: "Organize extracted data into a structured array or cell array with fields for spectra, m/z values, and spatial metadata. 5. Return the aggregated workspace structure containing all batch spectra in a"
- [readme] cdf files and available Matlab workspaces are provided for the roots that were analyzed: "cdf files and available Matlab workspaces are provided for the roots that were analyzed in Sama et al. 2025."
1---2name: batch-file-processing-orchestration-23description: Use when when you have multiple CDF imaging files (e.g., from mass spectrometry imaging scans of biological samples) that need to be read into a single Matlab workspace with consistent structure and metadata (spectral intensity, m/z arrays, spatial coordinates).4license: CC-BY-4.05---67# batch-file-processing-orchestration89## Summary1011Orchestrate batch ingestion of multiple CDF (NetCDF) imaging files into a structured Matlab workspace, aggregating spectral data, m/z values, and spatial coordinates for downstream mass spectrometry analysis. This skill enables efficient parallel or sequential processing of large imaging datasets typical in developmental mass spectrometry imaging workflows.1213## When to use1415When you have multiple CDF imaging files (e.g., from mass spectrometry imaging scans of biological samples) that need to be read into a single Matlab workspace with consistent structure and metadata (spectral intensity, m/z arrays, spatial coordinates). Typical trigger: a directory or file list containing 3+ CDF files destined for a linear or spatial imaging pipeline like DIMPLE.1617## When NOT to use1819- Input files are already merged into a single CDF or HDF5 file — use direct read instead.20- Spectral data are in non-imaging formats (e.g., single MS profiles without spatial metadata) — use simpler file I/O.21- You require real-time streaming or on-disk processing of files too large to aggregate in memory — use chunked or memory-mapped I/O instead.2223## Inputs2425- cell array of CDF file paths26- file list or directory containing CDF imaging files27- NetCDF/CDF file format spectra with embedded m/z and spatial metadata2829## Outputs3031- aggregated Matlab workspace structure with all batch spectra32- cell array or struct array with fields: spectra (intensity matrices), m/z values, spatial coordinates33- Matlab .mat file suitable for downstream DIMPLE pipeline analysis3435## How to apply3637Parse the input CDF file paths as a cell array or file list. Iterate through each CDF file using Matlab's NetCDF/CDF I/O functions (e.g., netcdf functions or dedicated CDF readers). For each file, extract spectral intensity arrays, m/z value vectors, and spatial coordinates (x, y position or linear position along the root/sample axis). Organize extracted data into a structured array or cell array with consistent fields (spectra, m/z, spatial metadata). Return the aggregated workspace structure containing all batch spectra in a format compatible with downstream pipeline analysis (e.g., DIMPLE filtering and segmentation). Validate that all spectra share the same m/z axis and that spatial metadata align with the sample geometry.3839## Related tools4041- **Matlab NetCDF/CDF I/O functions** (Read spectral intensity arrays, m/z values, and spatial metadata from individual CDF files)42- **batchcdfread function** (Custom batch orchestration function developed in Dr. Zare's lab; wraps NetCDF reads and aggregates output into workspace structure) — github.com/dickinsonlab/DIMPLE-code43- **DIMPLE pipeline** (Downstream consumer of aggregated batch workspace; performs filtering, segmentation, and linear imaging analysis) — github.com/dickinsonlab/DIMPLE-code4445## Evaluation signals4647- Output workspace contains one entry per input CDF file with no missing or truncated spectra.48- All spectra in the aggregated structure share the same m/z axis (vector length and values match across batch).49- Spatial coordinates are populated and consistent with sample geometry (e.g., linear positions 1 to N for N-scan root imaging).50- Output struct or cell array fields (spectra, m/z, spatial metadata) are present and non-empty for every batch member.51- Aggregated workspace can be directly loaded and processed by DIMPLE filtering/segmentation functions without schema errors.5253## Limitations5455- Requires sufficient RAM to hold all spectra in memory; large imaging datasets may exceed available workspace.56- Assumes all CDF files follow a consistent internal structure and metadata naming convention (e.g., all contain 'intensity', 'm/z', 'x', 'y' or 'position' fields); heterogeneous or malformed CDF files may cause read failures.57- No built-in error recovery or partial batch completion; a single corrupted CDF file halts the entire batch unless wrapped in error handling.58- Performance scales linearly with file count; no parallelization mentioned in the source material.5960## Evidence6162- [readme] batchcdfread function developed by Yifan Meng in Dr. Richard N. Zare's lab at Stanford University: "batchcdfread function developed by Yifan Meng in Dr. Richard N. Zare's lab at Stanford University"63- [other] Iterate through each CDF file and read the spectral data using Matlab's netCDF/CDF I/O functions. 3. Extract spectral intensity arrays, m/z values, and spatial coordinates: "Iterate through each CDF file and read the spectral data using Matlab's netCDF/CDF I/O functions. 3. Extract spectral intensity arrays, m/z values, and spatial coordinates (x, y, or linear position)"64- [other] Organize extracted data into a structured array or cell array with fields for spectra, m/z values, and spatial metadata. 5. Return the aggregated workspace structure containing all batch spectra in a format compatible with downstream DIMPLE pipeline analysis.: "Organize extracted data into a structured array or cell array with fields for spectra, m/z values, and spatial metadata. 5. Return the aggregated workspace structure containing all batch spectra in a"65- [readme] cdf files and available Matlab workspaces are provided for the roots that were analyzed: "cdf files and available Matlab workspaces are provided for the roots that were analyzed in Sama et al. 2025."
HolobiomicsLab