Gromov Wasserstein Similarity

gromov-wasserstein-similarity

Identifying Sub-networks in Neural Networks via Functionally Similar Representations — Tian Gao et al. (2024) (arXiv:2410.16484, 2024)

What this evaluates

Evaluates how well Gromov-Wasserstein distance captures functional similarity between neural network layer representations, enabling the identification of structural transitions and latent sub-networks across varying dimensionalities without task-specific supervision.

Datasets

• f_mod3 — total 3481; splits: train (-1), val (-1)

Metrics

• Gromov-Wasserstein distance (primary) — range: [0, ∞)
  • Computes the optimal transport cost between two probability distributions (layer activations) without requiring them to share the same space or dimensionality. It minimizes the discrepancy between pairwise distances in the source and target distributions.

Input / output format

Input: Intermediate activation tensors from specified neural network layers (e.g., transformer blocks, ResNet-9 modules).

Output: Scalar GW distance values between pairs of layer representations, indicating functional similarity.

Scoring recipe

def compute_gw_distance(X, Y, cost_X, cost_Y, reg=0.1):
    # X, Y: activation matrices (n_samples, d1), (m_samples, d2)
    # cost_X, cost_Y: pairwise distance matrices for X and Y
    # Uses standard GW optimal transport solver
    return gw_solver(X, Y, cost_X, cost_Y, reg)

Common pitfalls

• GW distance computation scales cubically with sample size, requiring subsampling for large batches.
• Results are sensitive to the choice of cost matrix (typically Euclidean) and regularization parameter.
• Layer activations must be properly normalized or centered before distance computation to avoid scale bias.

Evidence (verbatim from paper)

The paper proposes a task-agnostic, automated method to identify functionally distinct sub-networks in neural networks using Gromov-Wasserstein (GW) distance, which enables direct comparison of intermediate layer representations across different dimensionalities and distributions.

Citation

@misc{gao2024subnetworks,
  title={Identifying Sub-networks in Neural Networks via Functionally Similar Representations},
  author={Tian Gao et al. (2024)},
  year={2024},
  note={arXiv:2410.16484}
}

• arXiv: 2410.16484