chemical-informatics-data-format-conversion
Summary
Convert and standardize raw chemical structure notations (SMILES strings) into validated, deduplicated structural identifiers (standardized SMILES and InChI) through sequential file-to-file transformations. This skill is essential in natural products curation pipelines where structure data from heterogeneous sources must be normalized and validated before downstream chemical analysis.
When to use
When you have raw SMILES strings collected from multiple external databases that require standardization and deduplication before integration into a unified chemical structure database. Specifically, apply this skill when you need to convert interim/tables/0_original/structure/smiles.tsv.gz (raw SMILES from the gathering layer) into validated, unique structural identifiers that can be confidently linked to organism and reference data.
When NOT to use
- Input is already a validated, deduplicated unique structure table (e.g., from a trusted, pre-curated source).
- Raw data contains chemical structures in formats other than SMILES (e.g., MOL, SDF) without a prior conversion step to SMILES.
- You need to preserve all raw SMILES variants (including duplicates and minor stereoisomers) without merging them into canonical forms.
Inputs
- interim/tables/0_original/structure/smiles.tsv.gz (raw SMILES strings from gathering layer)
Outputs
- interim/tables/1_translated/structure/smiles.tsv.gz (standardized SMILES)
- interim/tables/1_translated/structure/unique.tsv.gz (deduplicated, validated structure identifiers)
How to apply
Execute two consecutive transformation steps: (1) Run smiles.py on the raw SMILES table to translate and standardize SMILES strings according to chemical validity rules, producing an intermediate standardized SMILES table. (2) Apply sanitizing.py to the translated SMILES to validate chemical structures, remove invalid entries (e.g., malformed SMILES that cannot be parsed), and deduplicate identical structures, generating the final unique structure table. The rationale is that raw SMILES from external sources often contain syntax errors, non-canonical forms, or duplicates; two-stage processing ensures both format compliance and chemical validity before structural identifiers are linked to organism and literature metadata.
Related tools
Examples
python scripts/smiles.py --input interim/tables/0_original/structure/smiles.tsv.gz --output interim/tables/1_translated/structure/smiles.tsv.gz && python scripts/sanitizing.py --input interim/tables/1_translated/structure/smiles.tsv.gz --output interim/tables/1_translated/structure/unique.tsv.gz
Evaluation signals
- All raw SMILES in the input file are successfully parsed and converted to standardized canonical form with no errors in smiles.py output.
- The number of unique structures in unique.tsv.gz is less than or equal to the number of translated SMILES (deduplication has occurred or is zero).
- Invalid SMILES strings (those that fail chemical validity checks in sanitizing.py) are documented and excluded from the unique structure table.
- Cross-validation: a random sample of structures in unique.tsv.gz can be converted back to SMILES and compared to the original raw SMILES to confirm standardization consistency.
- File schema validation: unique.tsv.gz contains expected columns (e.g., structure ID, canonical SMILES, InChI) with no null values in critical fields.
Limitations
- SMILES standardization is lossy for stereochemical detail when the input SMILES lacks explicit stereochemical annotation; minor stereoisomers will be merged into the canonical form.
- The sanitizing step removes structures that fail chemical validity rules, which may result in loss of data if source databases contain non-standard or ambiguous notations.
- Very large raw SMILES tables (>1M entries) may experience performance bottlenecks depending on Python environment memory and CPU resources.
- The skill requires that both smiles.py and sanitizing.py scripts are present and properly configured in the lotus-processor repository; modifications to these scripts will alter output behavior.
Evidence
- [methods] smiles.py converts raw SMILES from interim/tables/0_original/structure/smiles.tsv.gz, and sanitizing.py then produces the final standardized unique structure table at interim/tables/1_translated/structure/unique.tsv.gz: "The LOTUS processor applies two consecutive file-to-file transformations: smiles.py converts raw SMILES from interim/tables/0_original/structure/smiles.tsv.gz, and sanitizing.py then produces the"
- [methods] Load raw SMILES table from interim/tables/0_original/structure/smiles.tsv.gz using Python. Execute smiles.py to translate and standardise SMILES strings, generating interim/tables/1_translated/structure/smiles.tsv.gz. Load translated SMILES and apply sanitizing.py to validate chemical structures, remove invalid entries, and deduplicate to generate interim/tables/1_translated/structure/unique.tsv.gz.: "1. Load raw SMILES table from interim/tables/0_original/structure/smiles.tsv.gz using Python. 2. Execute smiles.py to translate and standardise SMILES strings, generating"
- [methods] 2_editing structure – SMILES processing, sanitization, and enrichment with workflow evidence showing file-to-file transformations: "220([interim/tables/0_original/structure/smiles.tsv.gz]) --> 221[[smiles.py]] --> 222([interim/tables/1_translated/structure/smiles.tsv.gz])"
- [readme] Python 3 is listed as a required tool in the README system requirements: "- R
- Python 3
- Java >= 17"
- [intro] LOTUS is described as a comprehensive collection of documented structure-organism pairs constructed from external databases requiring standardization: "LOTUS is a comprehensive collection of documented structure-organism pairs. Within the frame of current computational approaches in Natural Products research and related fields, these documented"
1---2name: chemical-informatics-data-format-conversion3description: Use when when you have raw SMILES strings collected from multiple external databases that require standardization and deduplication before integration into a unified chemical structure database. Specifically, apply this skill when you need to convert interim/tables/0_original/structure/smiles.tsv.4license: CC-BY-4.05---67# chemical-informatics-data-format-conversion89## Summary1011Convert and standardize raw chemical structure notations (SMILES strings) into validated, deduplicated structural identifiers (standardized SMILES and InChI) through sequential file-to-file transformations. This skill is essential in natural products curation pipelines where structure data from heterogeneous sources must be normalized and validated before downstream chemical analysis.1213## When to use1415When you have raw SMILES strings collected from multiple external databases that require standardization and deduplication before integration into a unified chemical structure database. Specifically, apply this skill when you need to convert interim/tables/0_original/structure/smiles.tsv.gz (raw SMILES from the gathering layer) into validated, unique structural identifiers that can be confidently linked to organism and reference data.1617## When NOT to use1819- Input is already a validated, deduplicated unique structure table (e.g., from a trusted, pre-curated source).20- Raw data contains chemical structures in formats other than SMILES (e.g., MOL, SDF) without a prior conversion step to SMILES.21- You need to preserve all raw SMILES variants (including duplicates and minor stereoisomers) without merging them into canonical forms.2223## Inputs2425- interim/tables/0_original/structure/smiles.tsv.gz (raw SMILES strings from gathering layer)2627## Outputs2829- interim/tables/1_translated/structure/smiles.tsv.gz (standardized SMILES)30- interim/tables/1_translated/structure/unique.tsv.gz (deduplicated, validated structure identifiers)3132## How to apply3334Execute two consecutive transformation steps: (1) Run smiles.py on the raw SMILES table to translate and standardize SMILES strings according to chemical validity rules, producing an intermediate standardized SMILES table. (2) Apply sanitizing.py to the translated SMILES to validate chemical structures, remove invalid entries (e.g., malformed SMILES that cannot be parsed), and deduplicate identical structures, generating the final unique structure table. The rationale is that raw SMILES from external sources often contain syntax errors, non-canonical forms, or duplicates; two-stage processing ensures both format compliance and chemical validity before structural identifiers are linked to organism and literature metadata.3536## Related tools3738- **smiles.py** (Translates and standardizes raw SMILES strings from interim/tables/0_original/structure/smiles.tsv.gz into canonical form) — https://github.com/lotusnprod/lotus-processor39- **sanitizing.py** (Validates chemical structures from translated SMILES, removes invalid entries, and deduplicates to generate the final unique structure table) — https://github.com/lotusnprod/lotus-processor40- **Python 3** (Execution environment for smiles.py and sanitizing.py scripts) — https://github.com/lotusnprod/lotus-processor4142## Examples4344```45python scripts/smiles.py --input interim/tables/0_original/structure/smiles.tsv.gz --output interim/tables/1_translated/structure/smiles.tsv.gz && python scripts/sanitizing.py --input interim/tables/1_translated/structure/smiles.tsv.gz --output interim/tables/1_translated/structure/unique.tsv.gz46```4748## Evaluation signals4950- All raw SMILES in the input file are successfully parsed and converted to standardized canonical form with no errors in smiles.py output.51- The number of unique structures in unique.tsv.gz is less than or equal to the number of translated SMILES (deduplication has occurred or is zero).52- Invalid SMILES strings (those that fail chemical validity checks in sanitizing.py) are documented and excluded from the unique structure table.53- Cross-validation: a random sample of structures in unique.tsv.gz can be converted back to SMILES and compared to the original raw SMILES to confirm standardization consistency.54- File schema validation: unique.tsv.gz contains expected columns (e.g., structure ID, canonical SMILES, InChI) with no null values in critical fields.5556## Limitations5758- SMILES standardization is lossy for stereochemical detail when the input SMILES lacks explicit stereochemical annotation; minor stereoisomers will be merged into the canonical form.59- The sanitizing step removes structures that fail chemical validity rules, which may result in loss of data if source databases contain non-standard or ambiguous notations.60- Very large raw SMILES tables (>1M entries) may experience performance bottlenecks depending on Python environment memory and CPU resources.61- The skill requires that both smiles.py and sanitizing.py scripts are present and properly configured in the lotus-processor repository; modifications to these scripts will alter output behavior.6263## Evidence6465- [methods] smiles.py converts raw SMILES from interim/tables/0_original/structure/smiles.tsv.gz, and sanitizing.py then produces the final standardized unique structure table at interim/tables/1_translated/structure/unique.tsv.gz: "The LOTUS processor applies two consecutive file-to-file transformations: smiles.py converts raw SMILES from interim/tables/0_original/structure/smiles.tsv.gz, and sanitizing.py then produces the"66- [methods] Load raw SMILES table from interim/tables/0_original/structure/smiles.tsv.gz using Python. Execute smiles.py to translate and standardise SMILES strings, generating interim/tables/1_translated/structure/smiles.tsv.gz. Load translated SMILES and apply sanitizing.py to validate chemical structures, remove invalid entries, and deduplicate to generate interim/tables/1_translated/structure/unique.tsv.gz.: "1. Load raw SMILES table from interim/tables/0_original/structure/smiles.tsv.gz using Python. 2. Execute smiles.py to translate and standardise SMILES strings, generating"67- [methods] 2_editing structure – SMILES processing, sanitization, and enrichment with workflow evidence showing file-to-file transformations: "220([interim/tables/0_original/structure/smiles.tsv.gz]) --> 221[[smiles.py]] --> 222([interim/tables/1_translated/structure/smiles.tsv.gz])"68- [readme] Python 3 is listed as a required tool in the README system requirements: "- R69- Python 370- Java >= 17"71- [intro] LOTUS is described as a comprehensive collection of documented structure-organism pairs constructed from external databases requiring standardization: "*LOTUS* is a comprehensive collection of documented structure-organism pairs. Within the frame of current computational approaches in Natural Products research and related fields, these documented"
HolobiomicsLab