Singer Identification Eval

singer-identification-eval

From Real to Cloned Singer Identification — Desblancs et al. (2024) (arXiv:2407.08647, 2024)

What this evaluates

Evaluates the ability of audio embedding and classification models to correctly identify the vocalist performing a song track. It specifically probes robustness to synthetic/deepfake voices and generalization across different music datasets and genre contexts.

Datasets

• Train — total 181989; splits: train (181989)
• Validation — total 8192; splits: val (8192)
• Closed — total 176141; splits: test (176141)
• FMA — total 11676; splits: test (11676)
• MTG — total 7710; splits: test (7710)
• Cloned — total 377; splits: test (377)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard top-1 classification accuracy: the proportion of tracks where the majority-voted predicted singer matches the ground-truth singer label.

Input / output format

Input: 6-second audio segments sampled at 16kHz, converted to 128-bin mel-spectrograms (128×240 tensor). At inference, multiple vocal segments per track are processed independently.

Output: Predicted singer ID per segment. Final track-level prediction is determined by majority voting over all segment predictions.

Scoring recipe

def score(predictions, gold):
    track_preds = []
    for seg_preds in predictions:
        votes = Counter(seg_preds)
        track_preds.append(votes.most_common(1)[0][0])
    correct = sum(1 for p, g in zip(track_preds, gold) if p == g)
    return correct / len(gold)

Common pitfalls

• Evaluating on a capella snippets instead of full tracks with instrumental backing, which differs from the paper's proposed authentic setup.
• Ignoring genre-specific performance drops (e.g., hip-hop, pop) and long-discography effects that significantly impact deepfake classification robustness.
• Using a single random train/test split per singer instead of the recommended 10-run protocol with fixed validation/test tracks for reproducibility.

Evidence (verbatim from paper)

For each dataset's test tracks, the final singer prediction is obtained using a majority vote scheme, where each segment with vocals is passed through the frozen embedding model and classification head. The singer with the most "votes" is then used as the track's final output. We report all our results using 10 runs. For both open datasets, we report results using all singers. On the other hand, for our 7500-singer closed set, we report results from 100 to 1000 classes.

Citation

@misc{desblancs2024real,
  title={From Real to Cloned Singer Identification},
  author={Desblancs et al. (2024)},
  year={2024},
  note={arXiv:2407.08647}
}

• arXiv: 2407.08647