Computational Reproducibility Verification

computational-reproducibility-verification

Summary

Verify that published computational results can be regenerated by re-executing the original analysis pipeline using deposited raw data and code. This skill validates the integrity of a computational workflow end-to-end, from raw inputs through intermediate processing steps to final outputs.

When to use

Apply this skill when you have access to both raw data (deposited in a repository like Zenodo) and analysis scripts (in a GitHub repository), and you need to confirm that the published figures, tables, or quantitative findings are reproducible. Use this especially for multi-step pipelines integrating heterogeneous data modalities (e.g., imaging and omics) where intermediate outputs must align with expected formats and dimensions before proceeding to the next stage.

When NOT to use

• Raw data or analysis scripts are not publicly available or are under restricted access.
• The published article does not provide sufficient detail on parameter values, software versions, or the execution order of analysis steps.
• Your goal is to conduct a novel analysis rather than validate existing results; use this skill only as a validation precursor, not as the primary analysis workflow.

Inputs

• Raw imaging data (IMC protein images, SIMS metabolite images)
• Raw mass spectrometry or imaging intensities in vendor or open formats
• Analysis scripts in Jupyter notebook or R Markdown format
• Repository README and dependency documentation (requirements.txt, environment.yml, package.json)

Outputs

• Regenerated processed image data (registered, segmented, or intensity-extracted)
• Regenerated single-cell feature tables (protein intensities, metabolite abundances, clustering assignments)
• Regenerated output figures and publication-ready visualizations
• Reproducibility report documenting intermediate outputs and deviations from published results

How to apply

Clone or download the analysis repository (e.g., from GitHub) and retrieve the raw data from the associated deposit (e.g., Zenodo DOI). Install all declared dependencies and runtime requirements (Python packages, R libraries, conda environments). Execute the analysis notebooks or scripts in the documented order, which typically proceeds through preprocessing, quality control, registration or alignment of data modalities, single-cell or regional segmentation, feature extraction, statistical testing, and visualization. At each stage, verify that intermediate outputs match expected schemas, file dimensions, and data types (e.g., image array shapes, feature table row/column counts, clustering assignments). Upon completion, directly compare the regenerated output figures and numerical results against those reported in the publication using visual inspection, quantitative metrics (pixel-level image comparison, table row matching), or statistical overlap assessments. Document any mismatches in data, parameters, or computational environment that could explain deviations.

Related tools

• ScSpaMet (End-to-end analysis pipeline for single-cell spatial metabolomics and protein profiling including image registration, segmentation, VAE joint embedding, and correlation analysis) — https://github.com/coskunlab/ScSpaMet
• Jupyter Notebook (Execution environment for running documented analysis workflows with sequential cell-based processing and in-place visualization)
• Zenodo deposit (Data repository storing raw imaging and mass spectrometry data associated with the published results) — https://doi.org/10.5281/zenodo.6784251

Examples

jupyter notebook notebooks/01_Processing_IMC_SIMS_images.ipynb && jupyter notebook notebooks/02_Registration_IMC_SIMS_images.ipynb && python -c "import pandas as pd; ref=pd.read_csv('reference_output.csv'); regen=pd.read_csv('regenerated_output.csv'); print((ref-regen).abs().max())"

Evaluation signals

• Intermediate outputs (segmentation masks, intensity feature tables, clustering assignments) match the expected dimensions and data types documented in the README.
• Regenerated output figures are visually identical or show negligible pixel-level differences (<5% deviation) compared to published figures.
• Numerical results (metabolite differences, distance metrics, correlation coefficients) reproduce within floating-point tolerance or published precision (e.g., ±0.01 for reported values).
• All analysis notebooks execute without errors and without requiring manual parameter adjustments beyond those listed in the repository documentation.
• Intermediate comparison outputs (e.g., VAE embeddings, trajectory projections) overlap substantially with published results when visualized in the same dimensional reduction space (e.g., >90% cell assignment consistency).

Limitations

• Reproducibility may fail if the computational environment (Python version, package versions, CUDA/GPU availability) differs significantly from the original execution environment; conda environments should be specified in the repository.
• Stochastic steps (random initialization in VAE, clustering algorithms) may produce slightly different outputs on re-execution even with fixed random seeds; tolerance thresholds for numerical comparison must be established a priori.
• Some intermediate file formats or directory structures may not be explicitly documented in the README, requiring inspection of notebook code or iterative debugging.
• Raw data availability may be restricted by data use agreements or institutional policies, preventing independent verification in certain contexts.

Evidence

• [other] The raw data and analysis scripts necessary for reproducing results are available in the ScSpaMet repository and at https://doi.org/10.5281/zenodo.6784251.: "The raw data and analysis scripts necessary for reproducing results are available in the ScSpaMet repository"
• [other] Workflow step 2 emphasizes installing dependencies and runtime requirements.: "Install all dependencies and runtime requirements specified in the repository documentation (e.g., Python packages, R libraries, or conda environments)."
• [other] Workflow verifies intermediate outputs match expected formats.: "Generate the output figure or table as defined in the pipeline configuration, verifying that intermediate outputs match expected formats and dimensions."
• [other] Final comparison against reference results confirms reproducibility.: "Compare the regenerated output against the reference result reported in the paper to confirm reproducibility."
• [readme] README details the multi-stage processing pipeline structure.: "01 Processing of IMC (protein) and SIMS (metabolite) images; 02 Registration of IMC and SIMS images for different imaging regions; 03 Single-cell level segmentation"