cross-database-entity-reconciliation
Summary
Integrate and deduplicate chemical entities (formulae, structures) sourced from multiple independent repositories (HMDB, ChEMBL, PubChem) by resolving cross-references and assigning unique identifiers, then enrich the unified entity set with biological relationship edges (BioEdges) from KEGG metabolic reaction pairs. This skill enables construction of a comprehensive, interconnected reference database suitable for high-precision mass spectrometry annotation.
When to use
Apply this skill when you have chemical entity records scattered across two or more public repositories (e.g., HMDB, ChEMBL, PubChem, KEGG) and need a single authoritative, deduplicated knowledge base indexed by a queryable identifier (e.g., m/z value or chemical formula). Use it specifically when downstream analysis (e.g., formula assignment in mass spectrometry imaging) requires both structural relationships (DBEdges) and metabolic/biological relationships (BioEdges) to rank and score candidate matches.
When NOT to use
- You already have a single, internally consistent chemical entity database with all required edges pre-computed; reconciliation would add overhead without new information.
- Your analysis is restricted to a single repository (e.g., only HMDB) and you do not require cross-source deduplication or biological relationship enrichment.
- Your input data is feature-by-sample peak intensity tables (metabolomics output) rather than raw chemical entity records; use formula assignment or annotation workflows instead.
Inputs
- Parsed chemical formulae and metadata tables from HMDB
- Parsed chemical formulae and metadata tables from ChEMBL
- Parsed chemical formulae and metadata tables from PubChem
- KEGG biological reactant pair records (linking formulae in metabolic reactions)
- Cross-reference mappings between repositories (e.g., InChIKey, PubChem CID to HMDB ID)
Outputs
- Unified KnownSet database comprising 2.8 million unique chemical formulae
- DBEdges: set of structural/chemical relationship edges linking formulae across HMDB, ChEMBL, and PubChem
- BioEdges: set of metabolic relationship edges linking formulae from KEGG reactant pairs
- Indexed, serialized database file (e.g., smart.db) with formula→m/z, formula→DBEdges, formula→BioEdges lookups
- Unique identifier mapping (formula ID → provenance source(s))
How to apply
First, retrieve and parse chemical formulae and metadata in bulk from each source repository (HMDB, ChEMBL, PubChem). Second, extract DBEdges by identifying structural and chemical relationships between formulae across the three databases, then deduplicate records and resolve cross-references using canonical identifiers or chemical structure matching. Third, independently retrieve KEGG biological reactant pairs and construct BioEdges that link formulae participating in the same metabolic reactions. Fourth, merge all unique formulae, DBEdges, and BioEdges into a unified directed graph structure, assigning a single unique identifier per formula and validating that all edge endpoints reference valid nodes. Finally, index the complete merged database by m/z (or other query key) and serialize to a database file (e.g., SQLite or pickle format) for efficient runtime lookups. At each deduplication step, retain provenance—recording which source repository contributed each entity—to support later scoring and filtering by database origin.
Related tools
- SMART (Orchestrates KnownSet database construction and uses the reconciled, indexed database for formula assignment in spatially-resolved mass spectrometry imaging via multiple linear regression scoring.) — https://github.com/bioinfo-ibms-pumc/SMART
Evaluation signals
- All 2.8 million formulae are present in the final merged database and each formula has a unique, canonical identifier with no duplicates.
- Every DBEdge references valid formula IDs on both endpoints; edge directionality and metadata (source repository) are preserved.
- Every BioEdge correctly links at least two formulae that appear together as reactants or products in a KEGG reaction; no edges reference non-existent formulae.
- Database index lookups by m/z (within ±PPM tolerance, default 5 ppm) return the expected formula candidates in reasonable runtime; no timeouts or missing results for reference m/z values.
- Provenance records show that duplicate formulae across repositories are merged (not replicated) with all source origins recorded; spot-check confirms cross-repository edges are present.
Limitations
- The raw SMART-database file size exceeds 1 Terabyte; most users receive a compressed or subset version (e.g., HMDB-only temporary database from Figshare). Full reconciliation requires significant storage and computational resources.
- Deduplication quality depends on the availability and accuracy of cross-repository mappings (e.g., InChIKey, CID mappings); discrepancies or missing mappings may result in false duplicates or false negatives.
- KEGG BioEdges are limited to reactions explicitly curated in KEGG; novel or tissue-specific metabolic pathways not yet in KEGG will not appear as BioEdges.
- The skill assumes formulae can be reliably parsed and matched across repositories; repositories with inconsistent or incomplete chemical metadata may introduce errors during reconciliation.
Evidence
- [other] Extract DBEdges by identifying structural and chemical relationships between formulae across the three databases, deduplicating and resolving cross-references.: "Extract DBEdges by identifying structural and chemical relationships between formulae across the three databases, deduplicating and resolving cross-references."
- [other] Retrieve KEGG biological reactant pairs and construct BioEdges that link formulae involved in metabolic reactions.: "Retrieve KEGG biological reactant pairs and construct BioEdges that link formulae involved in metabolic reactions."
- [other] Merge all formulae, DBEdges, and BioEdges into a unified graph structure, assigning unique identifiers and validating edge connectivity.: "Merge all formulae, DBEdges, and BioEdges into a unified graph structure, assigning unique identifiers and validating edge connectivity."
- [readme] SMART constructs a KnownSet database that comprise 2.8 million formulae interconnected by DBEdges sourced from repositories such as HMDB, ChEMBL, PubChem, and BioEdges from KEGG biological reactant pairs.: "SMART constructs a KnownSet database that comprise 2.8 million formulae interconnected by DBEdges sourced from repositories such as HMDB, ChEMBL, PubChem, and BioEdges from KEGG biological reactant"
- [readme] the raw SMART-database consists of huge number of formulae with their evidences, with database size exceeds 1 Terabyte: "the raw SMART-database consists of huge number of formulae with their evidences, with database size exceeds 1 Terabyte"
1---2name: cross-database-entity-reconciliation-23description: Use when you have chemical entity records scattered across two or more public repositories (e.g., HMDB, ChEMBL, PubChem, KEGG) and need a single authoritative, deduplicated knowledge base indexed by a queryable identifier (e.g., m/z value or chemical formula).4license: CC-BY-4.05---67# cross-database-entity-reconciliation89## Summary1011Integrate and deduplicate chemical entities (formulae, structures) sourced from multiple independent repositories (HMDB, ChEMBL, PubChem) by resolving cross-references and assigning unique identifiers, then enrich the unified entity set with biological relationship edges (BioEdges) from KEGG metabolic reaction pairs. This skill enables construction of a comprehensive, interconnected reference database suitable for high-precision mass spectrometry annotation.1213## When to use1415Apply this skill when you have chemical entity records scattered across two or more public repositories (e.g., HMDB, ChEMBL, PubChem, KEGG) and need a single authoritative, deduplicated knowledge base indexed by a queryable identifier (e.g., m/z value or chemical formula). Use it specifically when downstream analysis (e.g., formula assignment in mass spectrometry imaging) requires both structural relationships (DBEdges) and metabolic/biological relationships (BioEdges) to rank and score candidate matches.1617## When NOT to use1819- You already have a single, internally consistent chemical entity database with all required edges pre-computed; reconciliation would add overhead without new information.20- Your analysis is restricted to a single repository (e.g., only HMDB) and you do not require cross-source deduplication or biological relationship enrichment.21- Your input data is feature-by-sample peak intensity tables (metabolomics output) rather than raw chemical entity records; use formula assignment or annotation workflows instead.2223## Inputs2425- Parsed chemical formulae and metadata tables from HMDB26- Parsed chemical formulae and metadata tables from ChEMBL27- Parsed chemical formulae and metadata tables from PubChem28- KEGG biological reactant pair records (linking formulae in metabolic reactions)29- Cross-reference mappings between repositories (e.g., InChIKey, PubChem CID to HMDB ID)3031## Outputs3233- Unified KnownSet database comprising 2.8 million unique chemical formulae34- DBEdges: set of structural/chemical relationship edges linking formulae across HMDB, ChEMBL, and PubChem35- BioEdges: set of metabolic relationship edges linking formulae from KEGG reactant pairs36- Indexed, serialized database file (e.g., smart.db) with formula→m/z, formula→DBEdges, formula→BioEdges lookups37- Unique identifier mapping (formula ID → provenance source(s))3839## How to apply4041First, retrieve and parse chemical formulae and metadata in bulk from each source repository (HMDB, ChEMBL, PubChem). Second, extract DBEdges by identifying structural and chemical relationships between formulae across the three databases, then deduplicate records and resolve cross-references using canonical identifiers or chemical structure matching. Third, independently retrieve KEGG biological reactant pairs and construct BioEdges that link formulae participating in the same metabolic reactions. Fourth, merge all unique formulae, DBEdges, and BioEdges into a unified directed graph structure, assigning a single unique identifier per formula and validating that all edge endpoints reference valid nodes. Finally, index the complete merged database by m/z (or other query key) and serialize to a database file (e.g., SQLite or pickle format) for efficient runtime lookups. At each deduplication step, retain provenance—recording which source repository contributed each entity—to support later scoring and filtering by database origin.4243## Related tools4445- **SMART** (Orchestrates KnownSet database construction and uses the reconciled, indexed database for formula assignment in spatially-resolved mass spectrometry imaging via multiple linear regression scoring.) — https://github.com/bioinfo-ibms-pumc/SMART4647## Evaluation signals4849- All 2.8 million formulae are present in the final merged database and each formula has a unique, canonical identifier with no duplicates.50- Every DBEdge references valid formula IDs on both endpoints; edge directionality and metadata (source repository) are preserved.51- Every BioEdge correctly links at least two formulae that appear together as reactants or products in a KEGG reaction; no edges reference non-existent formulae.52- Database index lookups by m/z (within ±PPM tolerance, default 5 ppm) return the expected formula candidates in reasonable runtime; no timeouts or missing results for reference m/z values.53- Provenance records show that duplicate formulae across repositories are merged (not replicated) with all source origins recorded; spot-check confirms cross-repository edges are present.5455## Limitations5657- The raw SMART-database file size exceeds 1 Terabyte; most users receive a compressed or subset version (e.g., HMDB-only temporary database from Figshare). Full reconciliation requires significant storage and computational resources.58- Deduplication quality depends on the availability and accuracy of cross-repository mappings (e.g., InChIKey, CID mappings); discrepancies or missing mappings may result in false duplicates or false negatives.59- KEGG BioEdges are limited to reactions explicitly curated in KEGG; novel or tissue-specific metabolic pathways not yet in KEGG will not appear as BioEdges.60- The skill assumes formulae can be reliably parsed and matched across repositories; repositories with inconsistent or incomplete chemical metadata may introduce errors during reconciliation.6162## Evidence6364- [other] Extract DBEdges by identifying structural and chemical relationships between formulae across the three databases, deduplicating and resolving cross-references.: "Extract DBEdges by identifying structural and chemical relationships between formulae across the three databases, deduplicating and resolving cross-references."65- [other] Retrieve KEGG biological reactant pairs and construct BioEdges that link formulae involved in metabolic reactions.: "Retrieve KEGG biological reactant pairs and construct BioEdges that link formulae involved in metabolic reactions."66- [other] Merge all formulae, DBEdges, and BioEdges into a unified graph structure, assigning unique identifiers and validating edge connectivity.: "Merge all formulae, DBEdges, and BioEdges into a unified graph structure, assigning unique identifiers and validating edge connectivity."67- [readme] SMART constructs a KnownSet database that comprise 2.8 million formulae interconnected by DBEdges sourced from repositories such as HMDB, ChEMBL, PubChem, and BioEdges from KEGG biological reactant pairs.: "SMART constructs a KnownSet database that comprise 2.8 million formulae interconnected by DBEdges sourced from repositories such as HMDB, ChEMBL, PubChem, and BioEdges from KEGG biological reactant"68- [readme] the raw SMART-database consists of huge number of formulae with their evidences, with database size exceeds 1 Terabyte: "the raw SMART-database consists of huge number of formulae with their evidences, with database size exceeds 1 Terabyte"
HolobiomicsLab