Toyadmos Eval

toyadmos-eval

ToyADMOS: A Dataset of Miniature-Machine Operating Sounds for Anomalous Sound Detection — Koizumi et al. (2019) (arXiv:1908.03299, 2019)

What this evaluates

Evaluates unsupervised anomalous sound detection systems on miniature machine operating sounds. It probes the ability of models to learn normal acoustic patterns and identify deviations caused by mechanical faults or environmental variations.

Datasets

• ToyADMOS — total ?; splits: train (1000), test (-1); repo https://github.com/YumaKoizumi/ToyADMOS-dataset

Metrics

• AUC-ROC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve computed over file-level binary predictions derived from frame-level anomaly scores.

Input / output format

Input: 16 kHz audio wav-files, converted to 64-dimensional log-mel-amplitude spectra with 10 preceding and 10 succeeding time-frames concatenated as context.

Output: Binary file-level label (anomalous/normal) determined by whether the maximum frame-level reconstruction error exceeds a fixed threshold.

Scoring recipe

# For each test wav-file:
frame_scores = [reconstruction_error(frame) for frame in wav_file_frames]
file_score = max(frame_scores)
prediction = 1 if file_score > threshold else 0
# Compute AUC-ROC over all file predictions vs ground truth labels

Common pitfalls

• The toy-train sub-dataset mixes all 4 microphone channels into a single channel, unlike the others.
• File-level anomaly is declared if any single frame exceeds the threshold, not an average or majority vote.
• Environmental noise is mixed at a specific SNR (+10 dB) during preprocessing, which significantly affects baseline performance.

Evidence (verbatim from paper)

We calculated the anomaly scores on each time frame of all test wav-files. If the anomaly score exceeded the threshold even for one frame, the wav-file was determined to be anomalous. This system gave the area under the receiver operating characteristic curves of 0.874, 0.981, and 0.843 for the toy-car, toy-conveyor, and toy-train sub-datasets, respectively.

Citation

@misc{koizumi2019toyadmos,
  title={ToyADMOS: A Dataset of Miniature-Machine Operating Sounds for Anomalous Sound Detection},
  author={Koizumi et al. (2019)},
  year={2019},
  note={arXiv:1908.03299}
}

• arXiv: 1908.03299