Mammo Concept Alignment Eval

mammo-concept-alignment-eval

Mammo-CLIP Dissect: A Framework for Analysing Mammography Concepts in Vision-Language Models — Salahuddin et al. (2025) (arXiv:2509.21102, 2025)

What this evaluates

Evaluates how well vision-language models and CNNs capture clinically relevant mammography concepts at the neuron level. It quantifies concept coverage, alignment strength, and how domain-specific pretraining or task-specific fine-tuning shifts learned representations.

Datasets

• VinDR-Mammo — total ?; splits: test (-1); repo https://github.com/Suaiba/Mammo-CLIP-Dissect
• EMBED — total 84; splits: probe (84); repo https://github.com/Suaiba/Mammo-CLIP-Dissect

Metrics

• unique_concepts_captured (primary) — range: other
  • Counts the number of distinct mammography concepts whose mean similarity score to a target neuron exceeds a predefined threshold $ au$ (typically the layer-specific or global mean similarity). Overlapping concept terms are deduplicated to prevent double-counting.
• mean_similarity_score — range: [0, 1]
  • Similarity metric (e.g., cosine) between neuron activation vectors from the target model ($F_{target}$) and concept embeddings from the dissector model ($F_{dissector}$). Used to establish the threshold $ au$ for concept capture.

Input / output format

Input: Mammography images from probe datasets (VinDR-Mammo test set or EMBED subset) passed through a target CNN/VLM feature extractor ($F_{target}$) to obtain neuron activations, which are then compared against a fixed vocabulary of mammography concept embeddings using a vision-language model ($F_{dissector}$).

Output: Per-neuron similarity scores against all concepts in the vocabulary, plus a binary determination of whether each concept surpasses the threshold $ au$. Aggregated counts of unique concepts per layer and model variant.

Scoring recipe

def score_concept_coverage(neuron_activations, concept_embeddings, threshold_type='mean'):
    similarities = cosine_similarity(neuron_activations, concept_embeddings)
    if threshold_type == 'mean':
        tau = np.mean(similarities)
    else:
        tau = threshold_type
    captured = [c for c, s in zip(concepts, similarities) if s > tau]
    captured = deduplicate_overlapping_terms(captured)
    return len(captured)

Common pitfalls

• Threshold $ au$ is often set to the mean similarity, which varies significantly across layers and models, making cross-model comparisons highly sensitive to this choice.
• The concept vocabulary contains overlapping terms (e.g., 'extremely' vs 'extremely dense'); failure to explicitly deduplicate inflates or deflates coverage counts.
• Neuron-level responsiveness is heavily dependent on probe dataset composition (e.g., EMBED's implant subset vs VinDR-Mammo test set), limiting the generalizability of single-neuron examples.

Evidence (verbatim from paper)

First, we compare the mean similarity scores between neurons and concepts, used as thresholds, $ au$, at each layer within both $F_{target}$ models. This analysis is presented in Figure [4(a)]. From this figure, we can observe that M-Mammo-CLIP Dissect consistently has higher $ au$ values than G-Mammo-CLIP Dissect. Next, we analyse the number of unique concepts captured, surpassing the layer-specific threshold $ au$ across three representative layers.

Citation

@misc{salahuddin2025mammo,
  title={Mammo-CLIP Dissect: A Framework for Analysing Mammography Concepts in Vision-Language Models},
  author={Salahuddin et al. (2025)},
  year={2025},
  note={arXiv:2509.21102}
}

• arXiv: 2509.21102