Sound Localization Eval

sound-localization-eval

Learning Sound Localization Better From Semantically Similar Samples — Arda Senocak et al. (2022) (arXiv:2202.03007, 2022)

What this evaluates

Evaluates a model's ability to localize sound sources in audio-visual pairs by predicting spatial response maps or bounding boxes. It measures how effectively the model aligns audio signals with visual regions containing the corresponding sound, particularly testing robustness to semantically similar but mismatched cross-modal pairs.

Datasets

• VGGSound — total 200000; splits: train (200000)
• SoundNet-Flickr — total 144000; splits: train (144000)
• VGG-SS — total 5000; splits: test (5000)
• SoundNet-Flickr-Test — total 250; splits: test (250)

Metrics

• cIoU (primary) — range: [0, 1]
  • Center Intersection over Union. Computed as the area of intersection between the predicted and ground-truth bounding boxes divided by their union area, focusing on center alignment.
• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve. Measures the model's ability to rank positive sound-source locations higher than negatives across varying confidence thresholds.

Input / output format

Input: Center frames of video clips or still images paired with 3-second audio segments. Audio is converted to a 257×300 magnitude spectrogram. Both modalities are processed by ResNet18 backbones.

Output: Sound localization response maps or predicted bounding boxes indicating the spatial location of the sound source in the visual frame.

Scoring recipe

def compute_cIoU(pred_box, gt_box):
    inter_x1 = max(pred_box[0], gt_box[0])
    inter_y1 = max(pred_box[1], gt_box[1])
    inter_x2 = min(pred_box[2], gt_box[2])
    inter_y2 = min(pred_box[3], gt_box[3])
    inter_area = max(0, inter_x2 - inter_x1) * max(0, inter_y2 - inter_y1)
    union_area = (pred_box[2]-pred_box[0])*(pred_box[3]-pred_box[1]) + (gt_box[2]-gt_box[0])*(gt_box[3]-gt_box[1]) - inter_area
    return inter_area / union_area if union_area > 0 else 0.0

def compute_auc(scores, labels):
    return sklearn.metrics.roc_auc_score(labels, scores)

Common pitfalls

• Semantically similar but mismatched audio-visual pairs are often incorrectly treated as hard negatives in standard contrastive learning, which degrades localization performance.
• Randomly sampling hard positives introduces extreme noise and hurts training; they must be mined based on semantic similarity.
• Video-based models cannot be evaluated on SoundNet-Flickr because it only contains static image-audio pairs.

Evidence (verbatim from paper)

Table 2: Quantitative results on the SoundNet-Flickr test set. All models are trained and tested on the SoundNet-Flickr dataset.

Citation

@misc{senocak2022learning,
  title={Learning Sound Localization Better From Semantically Similar Samples},
  author={Arda Senocak et al. (2022)},
  year={2022},
  note={arXiv:2202.03007}
}

• arXiv: 2202.03007