Spatial Audio Motion Eval

spatial-audio-motion-eval

Spatial Audio Motion Understanding and Reasoning — Sridhar et al. (2025) (arXiv:2509.14666, 2025)

What this evaluates

Probes a model's ability to detect overlapping audio events in dynamic spatial recordings, estimate their direction of arrival and distance, and perform reasoning about moving sound sources.

Datasets

• STARS23 — total 30000; splits: train (30000), test (-1)
• FOA-MEIR Derived — total 3000; splits: test (3000)

Metrics

• F-score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall for frame-wise audio event detection, adopted from the DCASE 2025 Task 3 baseline protocol.
• DoA error — range: degrees
  • Mean absolute angular difference (in degrees) between predicted and ground-truth direction of arrival.
• Relative Distance Error — range: [0, 1]
  • Mean absolute difference between predicted and ground-truth source distance divided by ground-truth distance.

Input / output format

Input: 5-second stereo spatial audio clips containing up to three overlapping events, paired with text prompts for question answering.

Output: Frame-wise binary scores for audio tags, estimated DoA angles, estimated source distances, and textual answers (MCQ or Boolean) to spatial reasoning questions.

Scoring recipe

def evaluate(preds, gold):
    # preds, gold: dicts with keys 'tags', 'doa', 'dist'
    f1 = f1_score(gold['tags'], preds['tags'], average='macro')
    doa_err = np.mean(np.abs(gold['doa'] - preds['doa']))
    rde = np.mean(np.abs(gold['dist'] - preds['dist']) / gold['dist'])
    return {'F-score': f1, 'DoA error': doa_err, 'Relative Distance Error': rde}

Common pitfalls

• Baseline models were fine-tuned on additional synthetic data, but this protocol strictly uses only the public STARS23 training set for reproducibility.
• FOA recordings must be converted to stereo format before evaluation to align with the DCASE 2025 Task 3 baseline protocol.
• Generalization is evaluated on a custom dataset of 9 unseen events spatialized via Spatial Scaper, not on the original FOA-MEIR test split.

Evidence (verbatim from paper)

Evaluation is performed on the STARSS23 test set. To assess generalization, we built our 9 unseen isolated sound events dataset using dry source recordings from FOA-MEIR dataset[5]. We spatialize the audioclips using Spatial Scaper[13], generating 3k five-second stereo clips with up to three overlapping events per frame. FOA outputs are converted to stereo following[1]. We adopt the multi-ACCDDOA loss, evaluation metrics-F-score, DoA error, and Relative Distance Error, as well as feature extraction hyperparameters from[1].

Citation

@misc{sridhar2025spatial,
  title={Spatial Audio Motion Understanding and Reasoning},
  author={Sridhar et al. (2025)},
  year={2025},
  note={arXiv:2509.14666}
}

• arXiv: 2509.14666