Interquartile-range filtering for outlier rejection
Summary
Filter ensemble predictions by interquartile range (IQR) thresholds to identify and retain only high-confidence predictions, reducing model uncertainty and improving error metrics across prediction bins. This technique leverages multiple forward passes through a Monte-Carlo Dropout network to quantify prediction confidence and reject outliers.
When to use
When you have ensemble predictions (e.g., from Monte-Carlo Dropout inference with N ≥ 10 forward passes per input) and need to distinguish high-confidence from uncertain predictions before downstream analysis. Use this when model uncertainty correlates with prediction error and you can afford to discard a fraction of predictions in exchange for improved accuracy in retained samples.
When NOT to use
- Input predictions come from a single forward pass (no ensemble); IQR filtering requires multiple predictions per input to compute meaningful uncertainty.
- Ground-truth labels are not available; filtering requires validation against known targets to assess error reduction.
- The analysis requires 100% prediction coverage; IQR filtering discards predictions, reducing the dataset size and potentially introducing bias if uncertainty correlates with input characteristics rather than model confidence.
Inputs
- Ensemble predictions (N forward passes per input, typically N=10)
- Ground-truth labels or binning targets (e.g., reference Tanimoto scores)
- IQR threshold value (e.g., 0.025)
Outputs
- Filtered prediction set (subset of input predictions passing IQR threshold)
- Per-bin RMSE or error metric summary
- Sample retention rate and error reduction statistics
How to apply
For each input (e.g., spectrum pair), compute the median and interquartile range (IQR) from the N ensemble predictions. IQR is defined as the difference between the 75th and 25th percentiles. Retain only predictions where IQR falls below a chosen threshold (e.g., IQR < 0.025 for MS2DeepScore Tanimoto predictions). Stratify the retained predictions by ground-truth bins (e.g., by reference Tanimoto score ranges: 0.0–0.1, 0.1–0.2, etc.) and recompute per-bin error metrics (RMSE, MAE, or similar). Compare filtered and unfiltered RMSE to quantify the improvement; focus on bins where uncertainty is highest (typically extreme score ranges like <0.4 or >0.8 for similarity scores). The threshold is a hyperparameter that should be tuned based on the acceptable trade-off between sample retention rate and error reduction.
Related tools
- MS2DeepScore (Siamese neural network producing ensemble predictions via Monte-Carlo Dropout; IQR filtering applied to its Tanimoto score predictions) — https://github.com/matchms/ms2deepscore
- matchms (Spectrum data cleaning and preparation; used upstream to prepare spectra before MS2DeepScore inference) — https://github.com/matchms/matchms
- Monte-Carlo Dropout (Uncertainty quantification technique enabling multiple stochastic forward passes; provides the ensemble of predictions from which IQR is computed)
- scikit-learn (Percentile computation for IQR calculation; used for binning and stratified analysis)
Examples
from ms2deepscore.models import load_model
from ms2deepscore import MS2DeepScore
import numpy as np
model = load_model('ms2deepscore_model.pt')
ms2ds = MS2DeepScore(model)
# Run 10 forward passes with dropout enabled
predictions_ensemble = np.array([ms2ds.predict(spectrum_pairs) for _ in range(10)])
median_pred = np.median(predictions_ensemble, axis=0)
iqr_pred = np.percentile(predictions_ensemble, 75, axis=0) - np.percentile(predictions_ensemble, 25, axis=0)
# Filter by IQR < 0.025
filtered_mask = iqr_pred < 0.025
filtered_predictions = median_pred[filtered_mask]
Evaluation signals
- Per-bin RMSE for filtered predictions is lower than unfiltered baseline (expect 20–40% reduction in extreme bins, e.g., <0.4 and >0.8 Tanimoto ranges).
- Sample retention rate (fraction of predictions passing IQR threshold) is reported and trade-off with error reduction is quantified.
- IQR distribution shows clear separation between retained (low IQR) and rejected (high IQR) predictions; histograms should be unimodal or bimodal with a cutoff visible.
- Error reduction is most pronounced in extreme bins (Tanimoto <0.4 and >0.8); mid-range bins (0.5–0.7) may show slight degradation due to lower sample count.
- Threshold choice (e.g., IQR < 0.025) is justified by hyperparameter sweep showing optimal trade-off point on validation data.
Limitations
- IQR filtering assumes uncertainty (measured by IQR) is a reliable proxy for prediction error; systematic model biases unrelated to dropout variation will not be corrected.
- Threshold selection is dataset- and model-dependent; a threshold optimal for MS2DeepScore Tanimoto predictions may not generalize to other models or prediction targets.
- Discarding predictions reduces effective sample size, particularly in sparse bins; statistical power for bin-level analysis may be compromised.
- Extreme score ranges (Tanimoto <0.1 or >0.9) often have fewer ground-truth pairs in training data, so ensemble predictions in these regions may be inherently unreliable regardless of IQR.
- No guidance provided on computational requirements for N=10 forward passes; runtime scales linearly with ensemble size.
Evidence
- [other] Monte-Carlo Dropout filtering at IQR < 0.025 reduces average RMSE from 0.17 to 0.11 (34% improvement): "Monte-Carlo Dropout filtering at IQR < 0.025 reduces average RMSE from 0.17 to 0.11 (34% improvement)"
- [methods] At inference time, dropout was applied to all but the first layer of the base network. N = 10 embeddings were computed per spectrum pair: "At inference time, dropout was applied to all but the first layer of the base network. N = 10 embeddings were"
- [other] For each pair, compute the median prediction and interquartile range (IQR) from the 10 predictions: "For each pair, compute the median prediction and interquartile range (IQR) from the 10 predictions."
- [results] filtered out scores, according to increasingly stringent interquartile range (IQR) thresholds: "filtered out scores, according to increasingly stringent interquartile range (IQR) thresholds"
- [other] most significant RMSE reductions in low (< 0.4) and high (> 0.8) Tanimoto score ranges, while slightly increasing error in the mid score range (0.5–0.7): "most significant RMSE reductions in low (< 0.4) and high (> 0.8) Tanimoto score ranges, while slightly increasing error in the mid score range (0.5–0.7)"
- [intro] we achieve a root mean squared error for predicted Tanimoto scores of about 0.15 when run without uncertainty restrictions, and down to 0.1 with stronger restrictions on model uncertainty: "we achieve a root mean squared error for predicted Tanimoto scores of about 0.15 when run without uncertainty restrictions, and down to 0.1 with stronger restrictions on model uncertainty"
1---2name: interquartile-range-filtering-for-outlier-rejection3description: Use when when you have ensemble predictions (e.g., from Monte-Carlo Dropout inference with N ≥ 10 forward passes per input) and need to distinguish high-confidence from uncertain predictions before downstream analysis.4license: CC-BY-4.05---67# Interquartile-range filtering for outlier rejection89## Summary1011Filter ensemble predictions by interquartile range (IQR) thresholds to identify and retain only high-confidence predictions, reducing model uncertainty and improving error metrics across prediction bins. This technique leverages multiple forward passes through a Monte-Carlo Dropout network to quantify prediction confidence and reject outliers.1213## When to use1415When you have ensemble predictions (e.g., from Monte-Carlo Dropout inference with N ≥ 10 forward passes per input) and need to distinguish high-confidence from uncertain predictions before downstream analysis. Use this when model uncertainty correlates with prediction error and you can afford to discard a fraction of predictions in exchange for improved accuracy in retained samples.1617## When NOT to use1819- Input predictions come from a single forward pass (no ensemble); IQR filtering requires multiple predictions per input to compute meaningful uncertainty.20- Ground-truth labels are not available; filtering requires validation against known targets to assess error reduction.21- The analysis requires 100% prediction coverage; IQR filtering discards predictions, reducing the dataset size and potentially introducing bias if uncertainty correlates with input characteristics rather than model confidence.2223## Inputs2425- Ensemble predictions (N forward passes per input, typically N=10)26- Ground-truth labels or binning targets (e.g., reference Tanimoto scores)27- IQR threshold value (e.g., 0.025)2829## Outputs3031- Filtered prediction set (subset of input predictions passing IQR threshold)32- Per-bin RMSE or error metric summary33- Sample retention rate and error reduction statistics3435## How to apply3637For each input (e.g., spectrum pair), compute the median and interquartile range (IQR) from the N ensemble predictions. IQR is defined as the difference between the 75th and 25th percentiles. Retain only predictions where IQR falls below a chosen threshold (e.g., IQR < 0.025 for MS2DeepScore Tanimoto predictions). Stratify the retained predictions by ground-truth bins (e.g., by reference Tanimoto score ranges: 0.0–0.1, 0.1–0.2, etc.) and recompute per-bin error metrics (RMSE, MAE, or similar). Compare filtered and unfiltered RMSE to quantify the improvement; focus on bins where uncertainty is highest (typically extreme score ranges like <0.4 or >0.8 for similarity scores). The threshold is a hyperparameter that should be tuned based on the acceptable trade-off between sample retention rate and error reduction.3839## Related tools4041- **MS2DeepScore** (Siamese neural network producing ensemble predictions via Monte-Carlo Dropout; IQR filtering applied to its Tanimoto score predictions) — https://github.com/matchms/ms2deepscore42- **matchms** (Spectrum data cleaning and preparation; used upstream to prepare spectra before MS2DeepScore inference) — https://github.com/matchms/matchms43- **Monte-Carlo Dropout** (Uncertainty quantification technique enabling multiple stochastic forward passes; provides the ensemble of predictions from which IQR is computed)44- **scikit-learn** (Percentile computation for IQR calculation; used for binning and stratified analysis)4546## Examples4748```49from ms2deepscore.models import load_model50from ms2deepscore import MS2DeepScore51import numpy as np5253model = load_model('ms2deepscore_model.pt')54ms2ds = MS2DeepScore(model)5556# Run 10 forward passes with dropout enabled57predictions_ensemble = np.array([ms2ds.predict(spectrum_pairs) for _ in range(10)])58median_pred = np.median(predictions_ensemble, axis=0)59iqr_pred = np.percentile(predictions_ensemble, 75, axis=0) - np.percentile(predictions_ensemble, 25, axis=0)6061# Filter by IQR < 0.02562filtered_mask = iqr_pred < 0.02563filtered_predictions = median_pred[filtered_mask]64```6566## Evaluation signals6768- Per-bin RMSE for filtered predictions is lower than unfiltered baseline (expect 20–40% reduction in extreme bins, e.g., <0.4 and >0.8 Tanimoto ranges).69- Sample retention rate (fraction of predictions passing IQR threshold) is reported and trade-off with error reduction is quantified.70- IQR distribution shows clear separation between retained (low IQR) and rejected (high IQR) predictions; histograms should be unimodal or bimodal with a cutoff visible.71- Error reduction is most pronounced in extreme bins (Tanimoto <0.4 and >0.8); mid-range bins (0.5–0.7) may show slight degradation due to lower sample count.72- Threshold choice (e.g., IQR < 0.025) is justified by hyperparameter sweep showing optimal trade-off point on validation data.7374## Limitations7576- IQR filtering assumes uncertainty (measured by IQR) is a reliable proxy for prediction error; systematic model biases unrelated to dropout variation will not be corrected.77- Threshold selection is dataset- and model-dependent; a threshold optimal for MS2DeepScore Tanimoto predictions may not generalize to other models or prediction targets.78- Discarding predictions reduces effective sample size, particularly in sparse bins; statistical power for bin-level analysis may be compromised.79- Extreme score ranges (Tanimoto <0.1 or >0.9) often have fewer ground-truth pairs in training data, so ensemble predictions in these regions may be inherently unreliable regardless of IQR.80- No guidance provided on computational requirements for N=10 forward passes; runtime scales linearly with ensemble size.8182## Evidence8384- [other] Monte-Carlo Dropout filtering at IQR < 0.025 reduces average RMSE from 0.17 to 0.11 (34% improvement): "Monte-Carlo Dropout filtering at IQR < 0.025 reduces average RMSE from 0.17 to 0.11 (34% improvement)"85- [methods] At inference time, dropout was applied to all but the first layer of the base network. N = 10 embeddings were computed per spectrum pair: "At inference time, dropout was applied to all but the first layer of the base network. N = 10 embeddings were"86- [other] For each pair, compute the median prediction and interquartile range (IQR) from the 10 predictions: "For each pair, compute the median prediction and interquartile range (IQR) from the 10 predictions."87- [results] filtered out scores, according to increasingly stringent interquartile range (IQR) thresholds: "filtered out scores, according to increasingly stringent interquartile range (IQR) thresholds"88- [other] most significant RMSE reductions in low (< 0.4) and high (> 0.8) Tanimoto score ranges, while slightly increasing error in the mid score range (0.5–0.7): "most significant RMSE reductions in low (< 0.4) and high (> 0.8) Tanimoto score ranges, while slightly increasing error in the mid score range (0.5–0.7)"89- [intro] we achieve a root mean squared error for predicted Tanimoto scores of about 0.15 when run without uncertainty restrictions, and down to 0.1 with stronger restrictions on model uncertainty: "we achieve a root mean squared error for predicted Tanimoto scores of about 0.15 when run without uncertainty restrictions, and down to 0.1 with stronger restrictions on model uncertainty"
