cluster-validation-cdf-threshold-selection
Summary
Determine the optimal number of clusters in a dataset by computing consensus matrices across multiple trained models, calculating cumulative distribution functions (CDFs), and selecting the largest cluster number where the proportional change in area under the CDF exceeds a predefined threshold (e.g., Δk > 0.025). This approach validates clustering stability and prevents over-clustering.
When to use
When you have generated multiple biclusters or clusterings from ensemble models (e.g., 100 trained neural network models via cross-validation) and need to select an objective cluster number k without manual inspection. Particularly useful when clustering rows and columns of a matrix independently (e.g., microbes and metabolites from a bipartite feature attribution matrix) and seeking consensus across model runs.
When NOT to use
- Input is a single clustering result, not an ensemble of models — consensus methods require multiple independent runs to detect stable structure.
- You have prior domain knowledge strongly favoring a specific cluster number — this method is data-driven and may override expert judgment.
- The dataset has clear, well-separated clusters visible by visual inspection — simpler elbow methods or silhouette analysis may suffice.
Inputs
- Multiple trained clustering models or ensemble model weights (e.g., 100 neural networks trained via 10-fold cross-validation)
- Normalized feature matrix or attribution score matrix (e.g., S_i normalized by dividing by significant threshold and clipped to [-1, 1])
- Candidate range for cluster numbers k (e.g., k ∈ [2, 20])
Outputs
- Optimal cluster numbers k₁* (for rows) and k₂* (for columns)
- Consensus matrices for each k (average connectivity across ensemble models)
- CDF values and area-under-curve (AUC) for each k
- Elbow plot or area-change plot (Δk vs. k) for visual validation
- Final cluster assignments using k₁* and k₂*
How to apply
For each candidate cluster number k ranging from 2 to a maximum (e.g., 20), generate k-clusterings across all trained models (e.g., 100 models). Build a consensus matrix by averaging the connectivity matrices (binary adjacency indicating co-clustering) from all models. Calculate the cumulative distribution function (CDF) of each consensus matrix and compute the area under the CDF curve. Plot the area values against k and identify the largest k where the proportional change Δk = (area_k - area_k-1) / area_k-1 exceeds the threshold (e.g., 0.025). This peak-detection strategy identifies the elbow point where additional clusters add diminishing information gain. Apply this procedure separately to rows (e.g., microbes) and columns (e.g., metabolites) to obtain k₁* and k₂*.
Related tools
- Hierarchical clustering (Euclidean distance, complete linkage) (Initial clustering of rows and columns for each k value; used within the consensus clustering loop)
- Seaborn clustermap (Visualization tool for displaying hierarchical clusters and heatmaps during exploratory analysis)
- Consensus clustering (Core method that generates connectivity matrices across ensemble models and averages them to compute consensus)
- Python (NumPy, SciPy, Pandas) (Implementation language for computing CDFs, areas under curves, and proportional change calculations) — https://github.com/YDaiLab/MiMeNet
Evaluation signals
- Verify that the area-under-CDF plot shows a clear elbow or inflection point; a monotonically increasing or flat curve indicates poor cluster structure or inappropriate threshold.
- Confirm that selected k values (k₁* and k₂*) produce biologically or functionally meaningful clusters (e.g., metabolite modules enriched for disease state in statistical tests: Wilcoxon rank-sum, p < 0.05).
- Check that consensus matrices have high diagonal values (indicating strong within-cluster co-clustering) and low off-diagonal values for non-consensus assignments.
- Validate that the proportional change Δk transitions from above threshold to below threshold at k*, not scattered erratically across the range.
- Examine that cluster assignments remain stable across cross-validation folds — members should not shuffle dramatically between folds.
Limitations
- Method assumes ensemble models are sufficiently diverse; if all models converge to identical solutions, consensus matrices will be binary and CDF-based selection loses discriminative power.
- Threshold selection (e.g., Δk = 0.025) is somewhat arbitrary; sensitivity to this parameter is not formally analyzed in the article, and different datasets may require empirical tuning.
- Computational cost scales with the number of candidate k values and ensemble size; evaluating k ∈ [2, 20] across 100 models requires ~100 hierarchical clusterings per k.
- Method requires rows and columns to be clustered independently; it does not optimize a joint objective for both dimensions simultaneously, potentially missing biclusters that are tightly coupled.
- CDF-based area calculation is sensitive to the range and distribution of consensus values; heavily skewed or truncated distributions may produce misleading AUC comparisons.
Evidence
- [other] perform hierarchical clustering separately on microbe rows and metabolite columns using Euclidean distance and complete linkage with Seaborn's clustermap. 3. For each fixed cluster number k ranging from 2 to 20, generate a k-clustering and compute a consensus matrix by averaging connectivity matrices across all 100 trained models: "For each fixed cluster number k ranging from 2 to 20, generate a k-clustering and compute a consensus matrix by averaging connectivity matrices across all 100 trained models."
- [other] Calculate the cumulative distribution function (CDF) of each consensus matrix and compute the area under the CDF. 5. Determine optimal cluster numbers k₁* (microbes) and k₂* (metabolites) by selecting the largest k where the proportional change in area (Δk) exceeds 0.025 threshold: "Calculate the cumulative distribution function (CDF) of each consensus matrix and compute the area under the CDF. 5. Determine optimal cluster numbers k₁* (microbes) and k₂* (metabolites) by"
- [other] Bicluster the normalized score matrix S* using k₁* and k₂* to assign microbes and metabolites to their final module memberships.: "Bicluster the normalized score matrix S* using k₁* and k₂* to assign microbes and metabolites to their final module memberships."
- [other] Normalize the feature attribution score matrix S_i by dividing by the significant threshold score from background distribution and clip values to range [-1, 1].: "Normalize the feature attribution score matrix S_i by dividing by the significant threshold score from background distribution and clip values to range [-1, 1]."
- [other] identifying 8 modules of microbes and 8 modules of metabolites, with module feature values calculated as the average normalized abundance of members within each module.: "identifying 8 modules of microbes and 8 modules of metabolites, with module feature values calculated as the average normalized abundance of members within each module."
1---2name: cluster-validation-cdf-threshold-selection3description: Use when when you have generated multiple biclusters or clusterings from ensemble models (e.g., 100 trained neural network models via cross-validation) and need to select an objective cluster number k without manual inspection.4license: CC-BY-4.05---67# cluster-validation-cdf-threshold-selection89## Summary1011Determine the optimal number of clusters in a dataset by computing consensus matrices across multiple trained models, calculating cumulative distribution functions (CDFs), and selecting the largest cluster number where the proportional change in area under the CDF exceeds a predefined threshold (e.g., Δk > 0.025). This approach validates clustering stability and prevents over-clustering.1213## When to use1415When you have generated multiple biclusters or clusterings from ensemble models (e.g., 100 trained neural network models via cross-validation) and need to select an objective cluster number k without manual inspection. Particularly useful when clustering rows and columns of a matrix independently (e.g., microbes and metabolites from a bipartite feature attribution matrix) and seeking consensus across model runs.1617## When NOT to use1819- Input is a single clustering result, not an ensemble of models — consensus methods require multiple independent runs to detect stable structure.20- You have prior domain knowledge strongly favoring a specific cluster number — this method is data-driven and may override expert judgment.21- The dataset has clear, well-separated clusters visible by visual inspection — simpler elbow methods or silhouette analysis may suffice.2223## Inputs2425- Multiple trained clustering models or ensemble model weights (e.g., 100 neural networks trained via 10-fold cross-validation)26- Normalized feature matrix or attribution score matrix (e.g., S_i normalized by dividing by significant threshold and clipped to [-1, 1])27- Candidate range for cluster numbers k (e.g., k ∈ [2, 20])2829## Outputs3031- Optimal cluster numbers k₁* (for rows) and k₂* (for columns)32- Consensus matrices for each k (average connectivity across ensemble models)33- CDF values and area-under-curve (AUC) for each k34- Elbow plot or area-change plot (Δk vs. k) for visual validation35- Final cluster assignments using k₁* and k₂*3637## How to apply3839For each candidate cluster number k ranging from 2 to a maximum (e.g., 20), generate k-clusterings across all trained models (e.g., 100 models). Build a consensus matrix by averaging the connectivity matrices (binary adjacency indicating co-clustering) from all models. Calculate the cumulative distribution function (CDF) of each consensus matrix and compute the area under the CDF curve. Plot the area values against k and identify the largest k where the proportional change Δk = (area_k - area_k-1) / area_k-1 exceeds the threshold (e.g., 0.025). This peak-detection strategy identifies the elbow point where additional clusters add diminishing information gain. Apply this procedure separately to rows (e.g., microbes) and columns (e.g., metabolites) to obtain k₁* and k₂*.4041## Related tools4243- **Hierarchical clustering (Euclidean distance, complete linkage)** (Initial clustering of rows and columns for each k value; used within the consensus clustering loop)44- **Seaborn clustermap** (Visualization tool for displaying hierarchical clusters and heatmaps during exploratory analysis)45- **Consensus clustering** (Core method that generates connectivity matrices across ensemble models and averages them to compute consensus)46- **Python (NumPy, SciPy, Pandas)** (Implementation language for computing CDFs, areas under curves, and proportional change calculations) — https://github.com/YDaiLab/MiMeNet4748## Evaluation signals4950- Verify that the area-under-CDF plot shows a clear elbow or inflection point; a monotonically increasing or flat curve indicates poor cluster structure or inappropriate threshold.51- Confirm that selected k values (k₁* and k₂*) produce biologically or functionally meaningful clusters (e.g., metabolite modules enriched for disease state in statistical tests: Wilcoxon rank-sum, p < 0.05).52- Check that consensus matrices have high diagonal values (indicating strong within-cluster co-clustering) and low off-diagonal values for non-consensus assignments.53- Validate that the proportional change Δk transitions from above threshold to below threshold at k*, not scattered erratically across the range.54- Examine that cluster assignments remain stable across cross-validation folds — members should not shuffle dramatically between folds.5556## Limitations5758- Method assumes ensemble models are sufficiently diverse; if all models converge to identical solutions, consensus matrices will be binary and CDF-based selection loses discriminative power.59- Threshold selection (e.g., Δk = 0.025) is somewhat arbitrary; sensitivity to this parameter is not formally analyzed in the article, and different datasets may require empirical tuning.60- Computational cost scales with the number of candidate k values and ensemble size; evaluating k ∈ [2, 20] across 100 models requires ~100 hierarchical clusterings per k.61- Method requires rows and columns to be clustered independently; it does not optimize a joint objective for both dimensions simultaneously, potentially missing biclusters that are tightly coupled.62- CDF-based area calculation is sensitive to the range and distribution of consensus values; heavily skewed or truncated distributions may produce misleading AUC comparisons.6364## Evidence6566- [other] perform hierarchical clustering separately on microbe rows and metabolite columns using Euclidean distance and complete linkage with Seaborn's clustermap. 3. For each fixed cluster number k ranging from 2 to 20, generate a k-clustering and compute a consensus matrix by averaging connectivity matrices across all 100 trained models: "For each fixed cluster number k ranging from 2 to 20, generate a k-clustering and compute a consensus matrix by averaging connectivity matrices across all 100 trained models."67- [other] Calculate the cumulative distribution function (CDF) of each consensus matrix and compute the area under the CDF. 5. Determine optimal cluster numbers k₁* (microbes) and k₂* (metabolites) by selecting the largest k where the proportional change in area (Δk) exceeds 0.025 threshold: "Calculate the cumulative distribution function (CDF) of each consensus matrix and compute the area under the CDF. 5. Determine optimal cluster numbers k₁* (microbes) and k₂* (metabolites) by"68- [other] Bicluster the normalized score matrix S* using k₁* and k₂* to assign microbes and metabolites to their final module memberships.: "Bicluster the normalized score matrix S* using k₁* and k₂* to assign microbes and metabolites to their final module memberships."69- [other] Normalize the feature attribution score matrix S_i by dividing by the significant threshold score from background distribution and clip values to range [-1, 1].: "Normalize the feature attribution score matrix S_i by dividing by the significant threshold score from background distribution and clip values to range [-1, 1]."70- [other] identifying 8 modules of microbes and 8 modules of metabolites, with module feature values calculated as the average normalized abundance of members within each module.: "identifying 8 modules of microbes and 8 modules of metabolites, with module feature values calculated as the average normalized abundance of members within each module."
HolobiomicsLab