distance-clustering-in-high-dimensions
Summary
Cluster detected features across multiple LC-IMS-MS/MS samples by applying N-dimensional distance-based grouping in (mz, drift_time, retention_time) space with user-defined tolerances. This skill enables cross-sample feature matching and alignment by identifying features that occupy the same region across dimensions, accounting for instrument drift and calibration variation.
When to use
When you have detected feature tables from two or more LC-IMS-MS/MS samples and need to establish correspondence between features across samples (i.e., which features in sample A match which features in sample B). Apply this skill after feature detection but before consensus feature assignment, especially when dealing with instrument variation such as drift or calibration shifts that affect feature coordinates non-uniformly.
When NOT to use
- Input is already a pre-aligned consensus feature table with cluster assignments; re-clustering would duplicate work and risk degrading consensus coordinates.
- Single-sample analysis: cross-sample alignment requires at least two samples; use single-sample peak detection instead.
- Tolerance values are unknown or unmeasurable: clustering quality depends critically on appropriate dimension-specific tolerances; proceed only after empirical characterization of instrument variation.
Inputs
- Multiple detected feature tables (HDF5 .h5 format or mzML) from LC-IMS-MS/MS samples
- Feature columns: mz, drift_time, retention_time, intensity
- User-defined tolerance values for each dimension (mz_tol, drift_time_tol, retention_time_tol)
- Optional reference sample for coordinate normalization
Outputs
- Aligned feature table with columns: mz, drift_time, retention_time, intensity, cluster_id, sample_membership
- Consensus coordinates (mean or median) per cluster
- Sample-membership vector indicating which samples contributed to each cluster
- Row count equal to or exceeding input feature count (no feature loss)
How to apply
Load detected feature tables from all samples using deimos.load(), extracting mz, drift_time, retention_time, and intensity columns. Normalize feature coordinates across samples to a common reference frame (either a designated reference sample or a global coordinate system) to account for instrument variation. Apply N-dimensional distance-based clustering in (mz, drift_time, retention_time) space using configurable tolerances for each dimension to define matching windows; features within these tolerance windows in all dimensions are grouped into clusters. For each cluster, compute consensus coordinates (weighted mean or median) and assign cluster membership flags to track which samples contributed each feature. Filter clusters below a minimum sample-membership threshold if desired, then output the aligned feature table with original coordinates, consensus coordinates, cluster ID, and sample-membership vector.
Related tools
- DEIMoS (Python API and command-line tool providing N-dimensional distance-based clustering implementation and feature alignment workflow) — https://github.com/pnnl/deimos
- Python (Programming language in which DEIMoS is implemented; used for scripting clustering workflows)
- numpy (Numerical computing library used for N-dimensional distance calculations and coordinate normalization)
- ProteoWizard (Converts mzML and other mass spectrometry formats for loading into DEIMoS)
Examples
data_a = deimos.load('sample_a.h5', key='features', columns=['mz', 'drift_time', 'retention_time', 'intensity']); data_b = deimos.load('sample_b.h5', key='features', columns=['mz', 'drift_time', 'retention_time', 'intensity']); aligned = deimos.alignment.align([data_a, data_b], mz_tol=0.01, drift_time_tol=2.0, retention_time_tol=30.0)
Evaluation signals
- Output feature table contains all required columns: mz, drift_time, retention_time, cluster_id, sample_membership; validate schema before downstream analysis.
- Row count in output equals or exceeds input row count; no features are orphaned (each row assigned exactly one cluster ID).
- Cluster membership is consistent across dimensions: features assigned to the same cluster should fall within user-defined tolerances in mz, drift_time, and retention_time simultaneously.
- Sample-membership vector captures expected sample composition; clusters with too few samples (below threshold) are filtered out if threshold was specified.
- Consensus coordinates (weighted mean or median) per cluster lie within the convex hull of input feature coordinates in that cluster, indicating valid aggregation.
Limitations
- Clustering quality is sensitive to tolerance selection; poorly chosen tolerances will produce either over-fragmented clusters (tolerances too tight) or conflated features (tolerances too loose); tolerances must be empirically tuned to instrument and dataset characteristics.
- Non-linear coordinate variation (e.g., nonlinear drift across retention time or m/z) may not be fully captured by simple tolerance windows; DEIMoS uses reference-based and cross-sample alignment strategies to mitigate this, but extreme nonlinearity may require per-region tolerance adjustment.
- Clusters with only one contributing sample are retained in output; post-hoc filtering by sample-membership threshold is required if cross-sample features are needed.
- Algorithm assumes all samples are in a common coordinate system; if samples use different instrument calibrations or acquisition parameters, normalization to a reference frame is mandatory prior to clustering.
Evidence
- [other] N-dimensional distance-based clustering in (mz, drift_time, retention_time) space: "Apply N-dimensional distance-based clustering in (mz, drift_time, retention_time) space to group features across samples that occupy the same region; use configurable tolerances for each dimension to"
- [other] Feature coordinate normalization to account for instrument variation: "Normalize feature coordinates across samples to a common reference frame (either a designated reference sample or a global coordinate system). 3. Apply N-dimensional distance-based clustering in (mz,"
- [other] Consensus coordinate calculation and output structure: "For each cluster, compute consensus coordinates (weighted mean or median) and assign cluster membership flags to track which samples contributed each feature. 5. Output the aligned feature table with"
- [other] Validation criteria for aligned feature output: "Validation: confirm that output contains expected columns (mz, drift_time, retention_time, cluster_id, sample_membership), row count equals or exceeds input feature count, and no features are"
- [readme] N-dimensional algorithmic advantage for alignment confidence: "algorithm implementations simultaneously utilize all dimensions to (ii) increase alignment/feature matching confidence among datasets"
1---2name: distance-clustering-in-high-dimensions-23description: Use when when you have detected feature tables from two or more LC-IMS-MS/MS samples and need to establish correspondence between features across samples (i.e., which features in sample A match which features in sample B).4license: CC-BY-4.05---67# distance-clustering-in-high-dimensions89## Summary1011Cluster detected features across multiple LC-IMS-MS/MS samples by applying N-dimensional distance-based grouping in (mz, drift_time, retention_time) space with user-defined tolerances. This skill enables cross-sample feature matching and alignment by identifying features that occupy the same region across dimensions, accounting for instrument drift and calibration variation.1213## When to use1415When you have detected feature tables from two or more LC-IMS-MS/MS samples and need to establish correspondence between features across samples (i.e., which features in sample A match which features in sample B). Apply this skill after feature detection but before consensus feature assignment, especially when dealing with instrument variation such as drift or calibration shifts that affect feature coordinates non-uniformly.1617## When NOT to use1819- Input is already a pre-aligned consensus feature table with cluster assignments; re-clustering would duplicate work and risk degrading consensus coordinates.20- Single-sample analysis: cross-sample alignment requires at least two samples; use single-sample peak detection instead.21- Tolerance values are unknown or unmeasurable: clustering quality depends critically on appropriate dimension-specific tolerances; proceed only after empirical characterization of instrument variation.2223## Inputs2425- Multiple detected feature tables (HDF5 .h5 format or mzML) from LC-IMS-MS/MS samples26- Feature columns: mz, drift_time, retention_time, intensity27- User-defined tolerance values for each dimension (mz_tol, drift_time_tol, retention_time_tol)28- Optional reference sample for coordinate normalization2930## Outputs3132- Aligned feature table with columns: mz, drift_time, retention_time, intensity, cluster_id, sample_membership33- Consensus coordinates (mean or median) per cluster34- Sample-membership vector indicating which samples contributed to each cluster35- Row count equal to or exceeding input feature count (no feature loss)3637## How to apply3839Load detected feature tables from all samples using deimos.load(), extracting mz, drift_time, retention_time, and intensity columns. Normalize feature coordinates across samples to a common reference frame (either a designated reference sample or a global coordinate system) to account for instrument variation. Apply N-dimensional distance-based clustering in (mz, drift_time, retention_time) space using configurable tolerances for each dimension to define matching windows; features within these tolerance windows in all dimensions are grouped into clusters. For each cluster, compute consensus coordinates (weighted mean or median) and assign cluster membership flags to track which samples contributed each feature. Filter clusters below a minimum sample-membership threshold if desired, then output the aligned feature table with original coordinates, consensus coordinates, cluster ID, and sample-membership vector.4041## Related tools4243- **DEIMoS** (Python API and command-line tool providing N-dimensional distance-based clustering implementation and feature alignment workflow) — https://github.com/pnnl/deimos44- **Python** (Programming language in which DEIMoS is implemented; used for scripting clustering workflows)45- **numpy** (Numerical computing library used for N-dimensional distance calculations and coordinate normalization)46- **ProteoWizard** (Converts mzML and other mass spectrometry formats for loading into DEIMoS)4748## Examples4950```51data_a = deimos.load('sample_a.h5', key='features', columns=['mz', 'drift_time', 'retention_time', 'intensity']); data_b = deimos.load('sample_b.h5', key='features', columns=['mz', 'drift_time', 'retention_time', 'intensity']); aligned = deimos.alignment.align([data_a, data_b], mz_tol=0.01, drift_time_tol=2.0, retention_time_tol=30.0)52```5354## Evaluation signals5556- Output feature table contains all required columns: mz, drift_time, retention_time, cluster_id, sample_membership; validate schema before downstream analysis.57- Row count in output equals or exceeds input row count; no features are orphaned (each row assigned exactly one cluster ID).58- Cluster membership is consistent across dimensions: features assigned to the same cluster should fall within user-defined tolerances in mz, drift_time, and retention_time simultaneously.59- Sample-membership vector captures expected sample composition; clusters with too few samples (below threshold) are filtered out if threshold was specified.60- Consensus coordinates (weighted mean or median) per cluster lie within the convex hull of input feature coordinates in that cluster, indicating valid aggregation.6162## Limitations6364- Clustering quality is sensitive to tolerance selection; poorly chosen tolerances will produce either over-fragmented clusters (tolerances too tight) or conflated features (tolerances too loose); tolerances must be empirically tuned to instrument and dataset characteristics.65- Non-linear coordinate variation (e.g., nonlinear drift across retention time or m/z) may not be fully captured by simple tolerance windows; DEIMoS uses reference-based and cross-sample alignment strategies to mitigate this, but extreme nonlinearity may require per-region tolerance adjustment.66- Clusters with only one contributing sample are retained in output; post-hoc filtering by sample-membership threshold is required if cross-sample features are needed.67- Algorithm assumes all samples are in a common coordinate system; if samples use different instrument calibrations or acquisition parameters, normalization to a reference frame is mandatory prior to clustering.6869## Evidence7071- [other] N-dimensional distance-based clustering in (mz, drift_time, retention_time) space: "Apply N-dimensional distance-based clustering in (mz, drift_time, retention_time) space to group features across samples that occupy the same region; use configurable tolerances for each dimension to"72- [other] Feature coordinate normalization to account for instrument variation: "Normalize feature coordinates across samples to a common reference frame (either a designated reference sample or a global coordinate system). 3. Apply N-dimensional distance-based clustering in (mz,"73- [other] Consensus coordinate calculation and output structure: "For each cluster, compute consensus coordinates (weighted mean or median) and assign cluster membership flags to track which samples contributed each feature. 5. Output the aligned feature table with"74- [other] Validation criteria for aligned feature output: "Validation: confirm that output contains expected columns (mz, drift_time, retention_time, cluster_id, sample_membership), row count equals or exceeds input feature count, and no features are"75- [readme] N-dimensional algorithmic advantage for alignment confidence: "algorithm implementations simultaneously utilize all dimensions to (ii) increase alignment/feature matching confidence among datasets"
HolobiomicsLab 