Mmd JS Divergence

mmd_js_divergence

Unveiling Intrinsic Text Bias in Multimodal Large Language Models through Attention Key-Space Analysis — Zheng et al. (2025) (arXiv:2510.26721, 2025)

What this evaluates

Evaluates the geometric alignment between visual and textual attention key vectors in multimodal LLMs. It probes whether visual inputs occupy an out-of-distribution subspace relative to the text-centric key space learned during pretraining.

Datasets

• MMBench-CN — total ?; splits: test (-1)
• MMMU — total ?; splits: test (-1)

Metrics

• MMD (primary) — range: other
  • Maximum Mean Discrepancy measuring the distance between mean embeddings of image and text key vectors in a reproducing kernel Hilbert space. Computed on PCA-reduced key vectors.
• JS — range: [0, 1]
  • Jensen-Shannon divergence measuring the similarity between the probability distributions of image and text key vectors after PCA embedding. Values range from 0 (identical) to 1 (maximally different).

Input / output format

Input: Paired image and text tokens processed through the decoder layers of an MLLM (LLaVA-1.5-7B or Qwen2.5-VL-7B) to extract attention key vectors at specified layers.

Output: Scalar divergence scores (MMD and JS) quantifying the distributional gap between visual and textual key vectors, aggregated per layer and benchmark.

Scoring recipe

# Extract key vectors for image tokens (K_img) and text tokens (K_txt) at layer l
# Apply PCA to reduce dimensionality
K_img_pca = pca.fit_transform(K_img)
K_txt_pca = pca.fit_transform(K_txt)
# Compute MMD (using RBF kernel or similar as per paper)
mmd_score = compute_mmd(K_img_pca, K_txt_pca)
# Compute JS divergence (requires density estimation or histogram binning)
js_score = compute_js_divergence(K_img_pca, K_txt_pca)
return mmd_score, js_score

Common pitfalls

• MMD and JS values are sensitive to the choice of kernel and PCA dimensionality; results should not be compared across different preprocessing settings.
• The metric measures representation alignment, not downstream task accuracy; high divergence does not necessarily imply poor model performance.
• Intra-modality controls (Image vs Image, Text vs Text) must be computed to distinguish structural bias from measurement noise.

Evidence (verbatim from paper)

Using the PCA embeddings, we compute MMD- and JS-based modality divergence for each layer and benchmark. The aggregated statistical analysis confirms the core hypothesis by demonstrating a vast separation between cross-modality and intra-modality comparisons: The mean MMD for the Cross-Modality Gap (Image V.S. Text) is 0.408 (std=0.346), with the maximum divergence reaching 1.054 (LLaVA-1.5B, Layer 2).

Citation

@misc{zheng2025unveiling,
  title={Unveiling Intrinsic Text Bias in Multimodal Large Language Models through Attention Key-Space Analysis},
  author={Zheng et al. (2025)},
  year={2025},
  note={arXiv:2510.26721}
}

• arXiv: 2510.26721