Mit Saliency Benchmark Eval

mit-saliency-benchmark-eval

What do different evaluation metrics tell us about saliency models? — Bylinskii et al. (2016) (arXiv:1604.03605, 2016)

What this evaluates

Evaluates how well computational saliency models predict human visual attention on natural images. It probes the spatial accuracy and probabilistic alignment of predicted saliency maps against ground-truth eye-tracking fixations.

Datasets

• MIT Saliency Benchmark (MIT300) — total 300; splits: full (300); repo https://github.com/cvzoya/saliency

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic (ROC) curve. The saliency map is thresholded at multiple levels to create binary masks. True positive rate (TPR) is the proportion of ground-truth fixations covered by the mask. False positive rate (FPR) is the proportion of non-fixated image pixels covered. AUC is the integral of TPR over FPR.
• KL-divergence — range: [0, inf)
  • Kullback-Leibler divergence between the predicted saliency distribution and the ground-truth fixation distribution. Measures how the predicted map diverges from the human attention distribution.
• NSS — range: other
  • Normalized Scanpath Saliency. Computes the mean of the normalized saliency map values at ground-truth fixation locations. Measures how many standard deviations above the map's mean the fixations lie.
• CC — range: [-1, 1]
  • Pearson Correlation Coefficient between the predicted saliency map and the ground-truth fixation map. Measures linear correlation between predicted and observed attention distributions.
• EMD — range: other
  • Earth Mover's Distance. Measures the minimum cumulative distance required to transform the predicted saliency distribution into the ground-truth fixation distribution.

Input / output format

Input: A 2D saliency map (continuous distribution or discrete fixation locations) corresponding to one of the 300 natural images.

Output: A scalar metric score (e.g., AUC, KL, NSS, CC, EMD) quantifying the similarity or predictive power between the saliency map and ground-truth fixation data.

Scoring recipe

def compute_auc(saliency_map, gt_fixations, image_shape):
    # Normalize saliency map to [0, 1]
    # Threshold map at multiple levels to get binary masks
    tpr_list, fpr_list = [], []
    for threshold in thresholds:
        mask = (saliency_map >= threshold)
        tpr = sum(mask[gt_fixations]) / len(gt_fixations)
        fpr = sum(mask) / (image_shape[0] * image_shape[1])
        tpr_list.append(tpr)
        fpr_list.append(fpr)
    # Sort by FPR and compute area under curve
    auc = trapezoidal_rule(fpr_list, tpr_list)
    return auc

Common pitfalls

• Smoothing parameter selection for ground-truth fixation maps significantly affects metric scores; fixing it to 1 degree of visual angle is standard but not universal.
• Ambiguity between discrete fixation points and continuous fixation maps can lead to inconsistent evaluations across studies.
• Center bias in human fixations can artificially inflate scores for metrics that do not account for spatial priors.
• Task mismatch: models trained or evaluated on free-viewing data may perform poorly on visual search or other constrained tasks.

Evidence (verbatim from paper)

The AUC metric evaluates a saliency map’s predictive power by how many ground truth fixations it captures in successive level sets. To compute AUC, a saliency map (top left) is treated as a binary classifier of fixations at various threshold values (THRESH) and an ROC curve is swept out. Thresholding the saliency map produces the level sets in the bottom row. For each level set, the true positive rate is the proportion of fixations landing in the level set (top row, green points). The false positive rate is the proportion of image pixels in the level set not covered in fixations. We include 5 level sets corresponding to points on the ROC curve. The AUC score for the saliency map is the area under the ROC curve.

Citation

@misc{bylinskii2016saliency,
  title={What do different evaluation metrics tell us about saliency models?},
  author={Bylinskii et al. (2016)},
  year={2016},
  note={arXiv:1604.03605}
}

• arXiv: 1604.03605