Sonics Eval

sonics-eval

SONICS: Synthetic Or Not -- Identifying Counterfeit Songs — Md Awsafur Rahman et al. (2024) (arXiv:2408.14080, 2024)

What this evaluates

This benchmark evaluates a model's ability to distinguish between real human-recorded songs and AI-generated synthetic songs. It specifically probes long-range temporal dependency modeling by testing performance on both short (5s) and long (120s) audio clips, while also measuring generalization to unseen generation algorithms and singers.

Datasets

• SONICS — total 97000; splits: train (-1), valid (-1), test (-1); repo https://github.com/awsaf49/sonics

Metrics

• F1 score (primary) — range: [0, 1]
  • Binary average F1 score computed at a fixed decision threshold of 0.5. It balances precision and recall across the real (negative) and fake (positive) classes.
• Sensitivity — range: [0, 1]
  • True positive rate (recall) for the fake/AI class.
• Specificity — range: [0, 1]
  • True negative rate for the real/human class.

Input / output format

Input: Raw audio songs resampled to 16kHz, converted to mel-spectrograms (n_mels=128, hop_length=512, win_length=2048). Inputs are fixed to either 5 seconds (128x128 spectrogram) or 120 seconds (128x3744 spectrogram) via right-side zero-padding or middle cropping.

Output: Binary classification prediction (Real/Human vs Fake/AI) or probability score.

Scoring recipe

def compute_metrics(y_true, y_pred, threshold=0.5):
    y_pred_binary = (y_pred >= threshold).astype(int)
    tp = np.sum((y_pred_binary == 1) & (y_true == 1))
    fp = np.sum((y_pred_binary == 1) & (y_true == 0))
    fn = np.sum((y_pred_binary == 0) & (y_true == 1))
    tn = np.sum((y_pred_binary == 0) & (y_true == 0))
    precision = tp / (tp + fp + 1e-8)
    recall = tp / (tp + fn + 1e-8)
    f1 = 2 * precision * recall / (precision + recall + 1e-8)
    sensitivity = recall
    specificity = tn / (tn + fp + 1e-8)
    return {'F1': f1, 'Sensitivity': sensitivity, 'Specificity': specificity}

Common pitfalls

• Using Equal Error Rate (EER) as the primary metric can be misleading due to class imbalance in the dataset; the authors explicitly prioritize F1 score instead.
• Evaluating only on short audio clips (e.g., 5s) masks the model's ability to capture long-range temporal dependencies, which is the core capability this benchmark is designed to test.
• Failing to enforce strict train/valid-test separation for songs sharing the same (lyrics, style) inputs leads to data leakage and inflated performance on unseen algorithm/singer splits.

Evidence (verbatim from paper)

While existing methods (Zang et al., 2024b;a; Xie et al., 2024) use Equal Error Rate (EER) as a metric, we prioritize the F1 score (binary average, threshold = 0.5) as our primary metric due to EER's susceptibility to class imbalance. We also evaluate Sensitivity (Sens.) and Specificity (Spec.) to assess performance across fake and real classes.

Citation

@misc{rahman2024sonics,
  title={SONICS: Synthetic Or Not -- Identifying Counterfeit Songs},
  author={Md Awsafur Rahman et al. (2024)},
  year={2024},
  note={arXiv:2408.14080}
}

• arXiv: 2408.14080