Embedding Space Normalization And Geometry

embedding-space-normalization-and-geometry

Summary

Apply L2 normalization to fixed-dimension neural network embeddings and use element-wise geometric operations (e.g., Hadamard product) to combine normalized embedding spaces for downstream prediction tasks. This ensures bounded embedding magnitudes and enables interpretable feature interactions in multi-modal architectures.

When to use

When designing a Siamese or multi-branch neural architecture where two or more embedding streams (e.g., spectrum embeddings and formula embeddings) must be combined to produce a scalar logit or confidence score, and you need embeddings with controlled magnitude to enable stable element-wise operations and interpretable feature interactions.

When NOT to use

• Input embeddings are already sparse or categorical without continuous representation; L2 normalization is designed for dense continuous vectors.
• The task requires independent prediction from a single modality (e.g., spectrum-only or formula-only); Siamese geometry is unnecessary when fusion is not the goal.
• Atom counts or spectrum features are missing, incomplete, or inconsistent across the dataset; normalization cannot recover from missing modalities.

Inputs

• atom-count feature vectors (variable-length discrete feature counts for molecular elements)
• spectrum embeddings (fixed-dimension dense vectors, typically 512 dimensions post-normalization)
• batch of MS/MS spectra with precursor m/z and collision energy metadata

Outputs

• formula embeddings (512-dimensional L2-normalized vectors)
• rescoring logits (scalar confidence scores per spectrum–formula pair)
• refined formula candidate rankings with confidence scores

How to apply

Design each embedding branch (e.g., FormulaEncoder) to output fixed-dimension embeddings (e.g., 512 dimensions) with L2 normalization applied at the output layer to constrain the L2 norm to 1.0. In the fusion head (e.g., RescoreHead), compute the element-wise product (Hadamard product, ⊙) of the normalized spectrum embedding z_spec and normalized formula embedding z_form to produce a combined feature vector, then pass this through a final linear layer or pooling operation to generate the scalar logit. L2 normalization ensures that magnitude variations do not dominate the dot-product-like interaction; the element-wise product preserves per-dimension correlation information. Validate the forward pass with synthetic tensors matching expected batch sizes and embedding dimensions to confirm output shapes, normalization bounds (||z|| ≈ 1.0), and logit range.

Related tools

• FIDDLE (Deep learning framework implementing the FormulaEncoder and RescoreHead modules with L2 normalization for molecular formula prediction from MS/MS spectra) — https://github.com/JosieHong/FIDDLE
• msfiddle (PyPI package wrapping FIDDLE's Siamese architecture and providing CLI and Python API for batch prediction with normalized embedding inference) — https://github.com/josiehong/msfiddle
• PyTorch (Deep learning framework for implementing neural network modules (FormulaEncoder, RescoreHead) with L2 normalization via torch.nn.functional.normalize)

Examples

from torch.nn import Module, Linear, functional as F; embeddings = F.normalize(encoder(atom_counts), p=2, dim=1); logits = (embeddings * formula_embeddings).sum(dim=1, keepdim=True)

Evaluation signals

• Embedding L2 norms are approximately 1.0 (within ±0.01) after normalization for all batch samples.
• Forward pass produces scalar logit outputs with expected batch dimensions (batch_size, 1).
• Element-wise product (Hadamard product) of normalized embeddings has full rank (no dimension collapse) and non-zero gradient flow during backpropagation.
• Ranking of refined formula candidates by rescoring logits matches top-k accuracy benchmarks on held-out validation spectra (e.g., CASMI or NIST datasets).
• Ablation study confirming that removal of L2 normalization increases training instability or reduces top-k accuracy on external benchmarks.

Limitations

• L2 normalization discards magnitude information; if the magnitude of an embedding carries semantic meaning (e.g., confidence in the spectrum), that signal is lost.
• Element-wise products require embeddings of identical dimension; mismatched dimensions between spectrum and formula embeddings will cause runtime errors.
• Siamese architecture assumes both modalities (spectrum and formula) are present and well-represented; sparse or missing modalities cannot be processed.
• Rescoring logits are not calibrated probabilities; they should not be interpreted as absolute confidence without additional post-hoc calibration.
• The v2.0.0 rescore model replaces the legacy FDRNet class; checkpoints from earlier versions are not compatible and must be retrained.

Evidence

• [other] Design FormulaEncoder as a neural network layer that accepts atom-count feature vectors and produces 512-dimensional embeddings with L2 normalization.: "Design FormulaEncoder as a neural network layer that accepts atom-count feature vectors and produces 512-dimensional embeddings with L2 normalization"
• [other] RescoreHead computes element-wise product (⊙) of spectrum embedding z_spec and formula embedding z_form to produce a scalar logit output.: "Design RescoreHead as a module that computes element-wise product (⊙) of spectrum embedding z_spec and formula embedding z_form to produce a scalar logit output"
• [other] The v2.0.0 rescore model implements a Siamese architecture that replaces the removed FDRNet class, incorporating a FormulaEncoder to embed atom-count vectors and a RescoreHead that computes element-wise products of spectrum and formula embeddings to generate logits.: "The v2.0.0 rescore model implements a Siamese architecture that replaces the removed FDRNet class, incorporating a FormulaEncoder to embed atom-count vectors and a RescoreHead that computes"
• [other] Validate forward pass with synthetic atom-count and spectrum embedding tensors to confirm output shapes and normalization.: "Validate forward pass with synthetic atom-count and spectrum embedding tensors to confirm output shapes and normalization"
• [readme] Breaking change (v2.0.0): The rescore model has been redesigned (Siamese architecture), see details in CHANGELOG.md.: "Breaking change (v2.0.0): The rescore model has been redesigned (Siamese architecture)"