Transformer Input Preprocessing

Transformer Input Preprocessing

Summary

Encode discrete chemical formulae into fixed-length sinusoidal embedding vectors for input to a neural transformer architecture. This skill bridges raw chemical formula strings to learnable continuous representations suitable for energy-based scoring of formula–spectrum pairs.

When to use

Apply this skill when preparing chemical formulae (e.g., 'C6H12O6') as inputs to a transformer-based neural network for MS/MS spectrum scoring. Use it specifically when the transformer must rank multiple candidate formulae against an observed mass spectrum and element composition constraints are important; the sinusoidal basis preserves formula orthogonality across a dataset.

When NOT to use

• Formula candidates are already pre-embedded or vectorized by another method.
• The transformer input pipeline uses one-hot or other fixed categorical encodings incompatible with sinusoidal bases.
• The application requires negative ionization mode support; MIST-CF currently only supports positive mode adducts.

Inputs

• chemical formula strings (e.g., 'C6H12O6', 'H2O')
• element count dictionary or parsed formula representation
• embedding dimensionality parameter
• reference dataset of formulae (for validation of orthogonality and coverage)

Outputs

• fixed-length sinusoidal embedding vectors (float tensors)
• embedding orthogonality metrics (e.g., pairwise cosine distance distribution)
• embedding lookup table or encoder function
• dimensionality sufficiency report (coverage of formula diversity)

How to apply

Define a sinusoidal positional encoding scheme that applies sine and cosine basis functions to element counts and their positions within the formula string. Implement an encoder that maps discrete formula strings to fixed-length embedding vectors using this sinusoidal basis, ensuring all formulae map to the same dimensionality regardless of string length. Validate that embeddings maintain orthogonality across the training dataset and that the embedding dimension is sufficient to represent formula diversity. Generate an embedding lookup table or on-the-fly encoder so that any candidate formula generated by SIRIUS decomp can be rapidly converted to a transformer-compatible vector for scoring.

Related tools

• SIRIUS decomp (generates candidate chemical formulae for a given observed mass; output formulae are encoded by this skill into embeddings for transformer scoring) — https://bio.informatik.uni-jena.de/software/sirius/
• SCARF (prior work that developed the sinusoidal formula embedding technique adapted in MIST-CF) — https://arxiv.org/abs/2303.06470

Evaluation signals

• All formulae in the dataset map to the same fixed embedding dimensionality with no truncation or padding errors.
• Orthogonality check: pairwise cosine distances between embeddings of distinct formulae are well-distributed and not concentrated near 1.0 (which would indicate poor differentiation).
• Embedding coverage: the dimensionality is sufficient to represent the range of formula diversity in the training set without significant overlap or collision.
• Round-trip validation: a formula string can be encoded to an embedding and the embedding can be decoded back to the original formula without information loss.
• Consistency: identical formula strings always produce identical embeddings across multiple encoding runs.

Limitations

• The sinusoidal encoding scheme assumes a fixed maximum formula length or element set; formulae exceeding this may require truncation or special handling.
• Orthogonality is only validated on the training dataset; performance on out-of-distribution formulae (e.g., from untested organisms or synthetic pathways) is not guaranteed.
• MIST-CF currently supports only positive mode ionization; negative mode formulae cannot be embedded or scored.
• The choice of embedding dimensionality is a hyperparameter; insufficient dimensionality may cause formula collisions, while excessive dimensionality wastes compute.

Evidence

• [other] Define the sinusoidal positional encoding scheme for chemical formulae (sine and cosine functions applied to element counts and positions). Implement encoder that maps discrete chemical formula strings (e.g., 'C6H12O6') to fixed-length embedding vectors using the sinusoidal basis.: "Define the sinusoidal positional encoding scheme for chemical formulae (sine and cosine functions applied to element counts and positions). 2. Implement encoder that maps discrete chemical formula"
• [readme] Utilizing sinusoidal formula embeddings as developed in our previous work [SCARF]: "Utilizing sinusoidal formula embeddings as developed in our previous work [SCARF]"
• [other] Validate embedding orthogonality and dimensionality against formula diversity in a reference dataset.: "Validate embedding orthogonality and dimensionality against formula diversity in a reference dataset."
• [other] MIST-CF uses sinusoidal formula embeddings, a technique developed in prior work on SCARF, to encode chemical formulae for input to the formula transformer neural network architecture.: "MIST-CF uses sinusoidal formula embeddings, a technique developed in prior work on SCARF, to encode chemical formulae for input to the formula transformer neural network architecture."