Transformer Interpretability Eval

transformer-interpretability-eval

Transformer Interpretability Beyond Attention Visualization — Chefer et al. (2020) (arXiv:2012.09838, 2020)

What this evaluates

Evaluates the faithfulness and class-specificity of Transformer interpretability methods by measuring how well highlighted input features align with model predictions. It probes explanation quality through pixel/token masking, segmentation overlap, and rationale extraction accuracy.

Datasets

• ImageNet Validation (ILSVRC 2012) — total 50000; splits: test (50000)
• ImageNet-Segmentation — total 4276; splits: test (4276)
• Movie Reviews — total 2000; splits: train (1600), val (200), test (200)

Metrics

• AUC (Positive/Negative Perturbation) (primary) — range: percent
  • Area under the curve of mean top-1 accuracy versus percentage of masked pixels (10%-90%). Positive perturbation masks high-to-low relevance (lower AUC = better). Negative perturbation masks low-to-high relevance (higher AUC = better).
• Pixel Accuracy / mIoU / mAP — range: percent
  • Pixel accuracy after thresholding visualization by mean value. mIoU and mAP computed against ground truth segmentation masks. Higher is better.
• Token-F1 — range: percent
  • F1 score between predicted rationale (top-k tokens) and ground truth rationale tokens, evaluated for k=10 to 80 in steps of 10. No thresholding applied.

Input / output format

Input: Vision: sequence of 16x16 non-overlapping image patches plus a [CLS] token. Text: sequence of tokens up to 512 length plus a [CLS] token.

Output: Per-instance relevance/attributions map over input tokens or patches indicating their contribution to the predicted class.

Scoring recipe

def compute_perturbation_auc(vis, image, model, mode='positive'):
    scores = vis.flatten()
    indices = np.argsort(scores) if mode == 'positive' else np.argsort(scores)[::-1]
    accs = []
    for pct in range(10, 91, 10):
        mask = np.zeros_like(image)
        mask[indices[:int(pct/100*len(indices))]] = 1
        accs.append(mean_top1_accuracy(model, mask * image))
    return auc(np.arange(10, 91, 10), accs)

def compute_segmentation_metrics(vis, gt_mask):
    vis_thresh = (vis > vis.mean()).astype(float)
    pix_acc = np.mean(vis_thresh == gt_mask)
    iou = np.mean(intersection_over_union(vis_thresh, gt_mask))
    ap = average_precision(vis, gt_mask)
    return pix_acc, iou, ap

def compute_token_f1(vis, tokens, gt_rationale, k_range=range(10, 81, 10)):
    f1_scores = []
    for k in k_range:
        pred_rationale = set(tokens[np.argsort(vis)[-k:]])
        f1_scores.append(f1_score(gt_rationale, pred_rationale))
    return f1_scores

Common pitfalls

• Misinterpreting perturbation AUC direction: lower is better for positive perturbation (erasing important pixels drops accuracy), while higher is better for negative perturbation (erasing unimportant pixels maintains accuracy).
• Confusing predicted vs. target class evaluation: class-specific methods should be evaluated on the ground-truth target class, while class-agnostic methods show similar performance on both.
• NLP rationale extraction uses a fixed top-k token selection without adaptive thresholding, which can penalize methods that naturally assign lower relevance scores to fewer tokens.

Evidence (verbatim from paper)

The positive and negative perturbation tests follow a two-stage setting. First, a pre-trained network is used for extracting visualizations for the validation set of ImageNet. Second, we gradually mask out the pixels of the input image and measure the mean top-1 accuracy of the network. In positive perturbation, pixels are masked from the highest relevance to the lowest, while in the negative version, from lowest to highest. In positive perturbation, one expects to see a steep decrease in performance, which indicates that the masked pixels are important to the classification score. In negative perturbation, a good explanation would maintain the accuracy of the model, while removing pixels that are not related to the class. In both cases, we measure the area-under-the-curve (AUC), for erasing between 10% - 90% of the pixels.

Citation

@misc{chefer2020transformer,
  title={Transformer Interpretability Beyond Attention Visualization},
  author={Chefer et al. (2020)},
  year={2020},
  note={arXiv:2012.09838}
}

• arXiv: 2012.09838