Mimii Dg Eval

mimii-dg-eval

MIMII DG: Sound Dataset for Malfunctioning Industrial Machine Investigation and Inspection for Domain Generalization Task — Dohi et al. (2022) (arXiv:2205.13879, 2022)

What this evaluates

Evaluates domain generalization capabilities for anomalous sound detection by measuring how well models trained on a source domain of industrial machine sounds generalize to target domains with shifted operational parameters or background noise.

Datasets

• MIMII DG — total ?; splits: source (-1), target (-1)

Metrics

• AUC (primary) — range: percent
  • Area under the receiver operating characteristic curve. Calculated per domain, section, and machine type as: AUC = (1/(N_d^- * N_n^+)) * sum_{i=1}^{N_d^-} sum_{j=1}^{N_n^+} H(A_theta(x_j^+) - A_theta(x_i^-)), where H is the Heaviside step function, A_theta is the anomaly score, N_d^- is the number of normal test clips in domain d, and N_n^+ is the number of anomalous test clips in section n.

Input / output format

Input: Audio clips from industrial machines converted to log-Mel spectrograms (e.g., 640-dimensional vectors for autoencoder, concatenated frames for MobileNetV2).

Output: Anomaly score per audio clip.

Scoring recipe

def compute_auc(anomaly_scores_normal, anomaly_scores_anomalous):
    n_normal = len(anomaly_scores_normal)
    n_anomalous = len(anomaly_scores_anomalous)
    if n_normal == 0 or n_anomalous == 0:
        return 0.0
    correct_predictions = 0
    for score_normal in anomaly_scores_normal:
        for score_anomalous in anomaly_scores_anomalous:
            if score_anomalous > score_normal:
                correct_predictions += 1
    return correct_predictions / (n_normal * n_anomalous)

Common pitfalls

• The dataset evaluates domain generalization, so models must be trained only on source domain data and tested on both source and target domains without adaptation.
• AUC is computed separately for each domain, section, and machine type before averaging; reporting a single global AUC across all domains obscures the domain shift effect.
• Anomaly scores are threshold-independent for AUC, but the paper notes the detector should ideally work with the same threshold across domains for practical deployment.

Evidence (verbatim from paper)

We used the area under the receiver operating characteristic curve (AUC) for evaluation. Because the domain generalization task requires detecting anomalies even when the occurance of domain shifts can be difficult to detect, the anomaly detector is expected to work with the same threshold regardless of the domain. Therefore, we calculated the AUC using both the source and target domain data. Also, to evaluate the anomaly detection performance for each domain, the AUC was computed for each domain. The AUC for each domain, section, and machine type was calculated as

Citation

@misc{dohi2022mimiidg,
  title={MIMII DG: Sound Dataset for Malfunctioning Industrial Machine Investigation and Inspection for Domain Generalization Task},
  author={Dohi et al. (2022)},
  year={2022},
  note={arXiv:2205.13879}
}

• arXiv: 2205.13879