Pseldnets Seld Eval

pseldnets-seld-eval

PSELDNets: Pre-trained Neural Networks on a Large-scale Synthetic Dataset for Sound Event Localization and Detection — Jinbo Hu et al. (arXiv:2411.06399, 2024)

What this evaluates

Evaluates sound event localization and detection (SELD) performance on synthetic and real-world audio, measuring classification accuracy, localization precision, and overall detection quality across different network architectures and fine-tuning strategies.

Datasets

• synthetic-test-set — total ?; splits: test (-1)
• synthetic-training-set — total ?; splits: train (-1)
• Indoor Recordings — total ?; splits: test (-1)

Metrics

• ER20° — range: degree
  • Classification Error Rate at a 20° angular threshold. Lower values indicate better performance. Computed by comparing predicted and ground-truth DOA angles against a 20° tolerance.
• F20° — range: percent
  • F-score at a 20° angular threshold. Higher values indicate better performance. Balances precision and recall of correctly localized sound events within the 20° tolerance.
• LECD — range: degree
  • Localization Error Classification Distance. Lower values indicate better localization accuracy. Measures the average angular distance for correctly classified events.
• LRCD — range: [0, 1]
  • Localization Recall Classification Distance. Higher values indicate better recall of localized events. Computed as 1 minus the normalized localization error for detected events.
• SELD (primary) — range: [0, 1]
  • Composite SELD score. Lower values indicate better overall detection and localization performance. Aggregates classification and localization errors into a single metric following standard SELD evaluation protocols.

Input / output format

Input: Multi-channel or monophonic audio recordings (synthetic or real-world indoor recordings)

Output: SELD representations: ACCDOA, mACCCDOA, or EINV2 formats encoding sound class, onset/offset times, and azimuth/elevation coordinates

Scoring recipe

def compute_seld_metrics(preds, gold):
    # preds, gold: list of dicts with 'class', 'azimuth', 'elevation', 'onset', 'offset'
    # 1. Match predictions to ground truth within 20° angular threshold
    matches = []
    for p in preds:
        best_dist = 999
        for g in gold:
            dist = angular_distance(p['azimuth'], p['elevation'], g['azimuth'], g['elevation'])
            if dist < best_dist:
                best_dist = dist
        matches.append(best_dist)
    # 2. Compute ER20° and F20°
    tp = sum(1 for d in matches if d <= 20)
    er20 = 1 - (tp / max(len(gold), 1))
    f20 = tp / max(len(preds) + len(gold) - tp, 1)
    # 3. Compute LECD, LRCD, SELD (simplified aggregation)
    lecd = sum(d for d in matches if d <= 20) / max(tp, 1)
    lrcd = 1 - (lecd / 90)
    seld = (er20 + (1 - f20) + (1 - lrcd)) / 3
    return {'ER20°': er20, 'F20°': f20, 'LECD': lecd, 'LRCD': lrcd, 'SELD': seld}

Common pitfalls

• Threshold dependency: ER20° and F20° strictly use a 20° angular threshold, which may not reflect performance at other tolerances or in highly overlapping scenarios.
• Representation format mismatch: Comparing results across ACCDOA, mACCCDOA, and EINV2 requires careful handling of their distinct coordinate and classification encodings.
• Low-resource adaptation: AdapterBit and fine-tuning strategies may suffer from catastrophic interference if not properly constrained, affecting generalization to unseen rooms.

Evidence (verbatim from paper)

Firstly, the performance of PSELDNets is evaluated on synthetic-test-set, investigating various networks and SELD output formats. ... TABLE II: Results of various networks with the mACCDOA representations. ... ER20°↓ F20°↑ LECD↓ LRCD↑ SELD↓

Citation

@misc{hu2024pseldnets,
  title={PSELDNets: Pre-trained Neural Networks on a Large-scale Synthetic Dataset for Sound Event Localization and Detection},
  author={Jinbo Hu et al.},
  year={2024},
  note={arXiv:2411.06399}
}

• arXiv: 2411.06399