Protoscore Eval

protoscore-eval

From Confusion to Clarity: ProtoScore - A Framework for Evaluating Prototype-Based XAI — Monke et al. (2025) (arXiv:2511.08361, 2025)

What this evaluates

This benchmark evaluates the quality of prototype-based explainable AI (XAI) methods for time series and image data. It probes how well generated prototypes capture model behavior and data structure across nine interpretability properties, including correctness, consistency, continuity, and latent space cohesion.

Datasets

• ECG200 — total ?; splits: test (-1)

Metrics

• Total (primary) — range: [0, 1]
  • Arithmetic mean of the nine individual prototype quality properties (Correctness, Consistency, Continuity, Contrastivity, Covariate Complexity, Compactness, Confidence, Input Completeness, Cohesion of Latent Space), each scored on a [0, 1] scale where 0 indicates poor quality and 1 indicates highest quality.

Input / output format

Input: Trained prototype-based model, its generated prototypes, and the target dataset instances (time series or images).

Output: A dictionary containing scores for nine interpretability properties and a final Total score, each ranging from 0 to 1.

Scoring recipe

def compute_protoscore(prototypes, dataset):
    scores = {}
    for prop in ["CR", "CS", "CN", "CT", "CC", "CP", "CF", "IC", "CLS"]:
        scores[prop] = evaluate_property(prop, prototypes, dataset) # Returns 0.0-1.0
    scores["Total"] = sum(scores.values()) / len(scores)
    return scores

Common pitfalls

• High validation loss does not necessarily indicate poor prototype quality; predictive performance and explanation quality are often decoupled.
• Dimensionality reduction techniques like UMAP distort true distances, so visual latent space plots should not be used for quantitative metric comparisons.
• Trade-offs exist between properties (e.g., high Contrastivity reduces Continuity), so optimizing a single metric may degrade others.

Evidence (verbatim from paper)

The total score averages the individual scores with equal weighting to summarize the prototype quality. Alongside, the mean squared error (MSE) validation loss for each model is given, which provides context for the model performance. The methods are sorted by descending validation loss. Note that a score of 0 indicates poor quality of the assessed prototypes, and a score of 1 represents the highest quality possible.

Citation

@misc{monke2025protoscore,
  title={From Confusion to Clarity: ProtoScore - A Framework for Evaluating Prototype-Based XAI},
  author={Monke et al. (2025)},
  year={2025},
  note={arXiv:2511.08361}
}

• arXiv: 2511.08361