filter-parameter-validation
Summary
Validate that filter parameters (numeric ranges, group inclusion/exclusion criteria, thresholds) are correctly specified and enforced before applying them to feature tables. This skill ensures filter logic operates on well-formed inputs and produces reproducible, auditable results.
When to use
Apply this skill when reconstructing or modifying dashboard filters (e.g., fold-change, phenotype score) that accept user-specified numeric bounds or group membership criteria, especially when those filters have not been formally tested or when filter behavior needs to be verified against a reference implementation.
When NOT to use
- Input is already a feature table that has been filtered by a previously validated pipeline—skip re-validation unless filter logic has changed.
- Filter parameters are hard-coded constants in a mature, long-running production system—validate only if source code is modified or new data types are introduced.
- Numeric bounds or group criteria come from an external, certified data source with its own validation guarantees.
Inputs
- Feature table (numeric matrix with features as rows, samples/groups as columns)
- Fold-change or phenotype-score filter specification (minimum bound, maximum bound, group list, inclusion/exclusion flag)
- Group metadata (sample-to-group mapping)
Outputs
- Validated filter parameter set (confirmed numeric and logical consistency)
- Filtered feature table (rows retained after applying validated bounds and group criteria)
- pytest test report (pass/fail on row count, threshold compliance, and group logic)
How to apply
Define the filter interface to accept minimum and maximum numeric bounds (e.g., fold-change thresholds) and a group specification (include/exclude list). Implement the filter calculation logic that compares feature intensities or scores across specified groups. Before applying the filter to a production feature table, use pytest to validate: (1) that output row count matches the expected filtered set size, (2) that all retained features numerically satisfy the fold-change threshold or other metric, and (3) that group inclusion/exclusion logic is correctly applied. This validation step prevents silent failures where filters silently pass wrong rows or drop correct ones.
Related tools
- pytest (Unit testing framework used to validate filter correctness: verify output row count, confirm all retained features meet the fold-change threshold, and check group inclusion/exclusion logic.) — https://github.com/pytest-dev/pytest
- fermo_core (Core metabolomics data processing library that implements fold-change calculation and filtering logic for the FERMO dashboard.) — https://github.com/fermo-metabolomics/fermo_core
Evaluation signals
- Output feature table row count exactly matches the expected count of features passing both the numeric threshold (fold-change range) and group criteria.
- All retained features in the output table have fold-change values (or phenotype scores) within the specified minimum and maximum bounds when calculated across the specified groups.
- Group inclusion/exclusion filtering correctly filters features: if 'include' is specified, only features from those groups appear; if 'exclude', no features from excluded groups appear.
- pytest assertions pass without error for all test cases covering boundary conditions (e.g., features at exactly the min/max threshold, single-group filters, empty group lists).
- Spot-check: manually verify 2–3 features from the filtered output table against the original unfiltered table to confirm the fold-change calculation and group selection are correct.
Limitations
- The README provides only header content and overview; the technical specification of fold-change calculation (e.g., whether it is log2-fold-change, arithmetic ratio, or another metric) is not documented in the provided materials.
- Filter validation assumes that input group metadata is correctly formed and complete; validation does not detect missing or mislabeled samples.
- pytest validation is unit-level and does not account for edge cases in upstream data preparation (e.g., zero intensities, missing values) that may cause fold-change to be undefined or produce NaN/Inf.
Evidence
- [other] Use pytest to validate filter correctness: verify output row count matches expected filtered set, check that all retained features meet the fold-change threshold, and confirm group inclusion/exclusion logic is correctly applied.: "Use pytest to validate filter correctness: verify output row count matches expected filtered set, check that all retained features meet the fold-change threshold, and confirm group"
- [readme] FERMO integrates metabolomics data with orthogonal data such as phenotype information for rapid, biochemometric, hypothesis-driven prioritization.: "FERMO integrates metabolomics data with orthogonal data such as phenotype information for rapid, biochemometric, hypothesis-driven prioritization."
- [other] Define the filter interface to accept minimum and maximum fold-change bounds and a group specification (include/exclude list).: "Define the filter interface to accept minimum and maximum fold-change bounds and a group specification (include/exclude list)."
- [other] Implement the fold-change calculation logic comparing feature intensities across specified groups.: "Implement the fold-change calculation logic comparing feature intensities across specified groups."
1---2name: filter-parameter-validation3description: Use when reconstructing or modifying dashboard filters (e.4license: CC-BY-4.05---67# filter-parameter-validation89## Summary1011Validate that filter parameters (numeric ranges, group inclusion/exclusion criteria, thresholds) are correctly specified and enforced before applying them to feature tables. This skill ensures filter logic operates on well-formed inputs and produces reproducible, auditable results.1213## When to use1415Apply this skill when reconstructing or modifying dashboard filters (e.g., fold-change, phenotype score) that accept user-specified numeric bounds or group membership criteria, especially when those filters have not been formally tested or when filter behavior needs to be verified against a reference implementation.1617## When NOT to use1819- Input is already a feature table that has been filtered by a previously validated pipeline—skip re-validation unless filter logic has changed.20- Filter parameters are hard-coded constants in a mature, long-running production system—validate only if source code is modified or new data types are introduced.21- Numeric bounds or group criteria come from an external, certified data source with its own validation guarantees.2223## Inputs2425- Feature table (numeric matrix with features as rows, samples/groups as columns)26- Fold-change or phenotype-score filter specification (minimum bound, maximum bound, group list, inclusion/exclusion flag)27- Group metadata (sample-to-group mapping)2829## Outputs3031- Validated filter parameter set (confirmed numeric and logical consistency)32- Filtered feature table (rows retained after applying validated bounds and group criteria)33- pytest test report (pass/fail on row count, threshold compliance, and group logic)3435## How to apply3637Define the filter interface to accept minimum and maximum numeric bounds (e.g., fold-change thresholds) and a group specification (include/exclude list). Implement the filter calculation logic that compares feature intensities or scores across specified groups. Before applying the filter to a production feature table, use pytest to validate: (1) that output row count matches the expected filtered set size, (2) that all retained features numerically satisfy the fold-change threshold or other metric, and (3) that group inclusion/exclusion logic is correctly applied. This validation step prevents silent failures where filters silently pass wrong rows or drop correct ones.3839## Related tools4041- **pytest** (Unit testing framework used to validate filter correctness: verify output row count, confirm all retained features meet the fold-change threshold, and check group inclusion/exclusion logic.) — https://github.com/pytest-dev/pytest42- **fermo_core** (Core metabolomics data processing library that implements fold-change calculation and filtering logic for the FERMO dashboard.) — https://github.com/fermo-metabolomics/fermo_core4344## Evaluation signals4546- Output feature table row count exactly matches the expected count of features passing both the numeric threshold (fold-change range) and group criteria.47- All retained features in the output table have fold-change values (or phenotype scores) within the specified minimum and maximum bounds when calculated across the specified groups.48- Group inclusion/exclusion filtering correctly filters features: if 'include' is specified, only features from those groups appear; if 'exclude', no features from excluded groups appear.49- pytest assertions pass without error for all test cases covering boundary conditions (e.g., features at exactly the min/max threshold, single-group filters, empty group lists).50- Spot-check: manually verify 2–3 features from the filtered output table against the original unfiltered table to confirm the fold-change calculation and group selection are correct.5152## Limitations5354- The README provides only header content and overview; the technical specification of fold-change calculation (e.g., whether it is log2-fold-change, arithmetic ratio, or another metric) is not documented in the provided materials.55- Filter validation assumes that input group metadata is correctly formed and complete; validation does not detect missing or mislabeled samples.56- pytest validation is unit-level and does not account for edge cases in upstream data preparation (e.g., zero intensities, missing values) that may cause fold-change to be undefined or produce NaN/Inf.5758## Evidence5960- [other] Use pytest to validate filter correctness: verify output row count matches expected filtered set, check that all retained features meet the fold-change threshold, and confirm group inclusion/exclusion logic is correctly applied.: "Use pytest to validate filter correctness: verify output row count matches expected filtered set, check that all retained features meet the fold-change threshold, and confirm group"61- [readme] FERMO integrates metabolomics data with orthogonal data such as phenotype information for rapid, biochemometric, hypothesis-driven prioritization.: "FERMO integrates metabolomics data with orthogonal data such as phenotype information for rapid, biochemometric, hypothesis-driven prioritization."62- [other] Define the filter interface to accept minimum and maximum fold-change bounds and a group specification (include/exclude list).: "Define the filter interface to accept minimum and maximum fold-change bounds and a group specification (include/exclude list)."63- [other] Implement the fold-change calculation logic comparing feature intensities across specified groups.: "Implement the fold-change calculation logic comparing feature intensities across specified groups."
HolobiomicsLab