Songbsab Eval

songbsab-eval

SongBsAb: A Dual Prevention Approach against Singing Voice Conversion based Illegal Song Covers — Guangke Chen et al. (2024) (arXiv:2401.17133, 2024)

What this evaluates

Evaluates the effectiveness of an adversarial perturbation method (SongBsAb) designed to prevent illegal singing voice conversion. It probes the method's ability to disrupt singer identity and lyrical fidelity in converted audio while maintaining high audio quality and imperceptibility.

Datasets

• OpenSinger — total 43075; splits: test (1000)
• NUS-48E — total 510; splits: test (1000)

Metrics

• Identity Similarity — range: [-1, 1]
  • Cosine similarity between the centroid identity feature of the target singer and the identity feature of the output singing voice, extracted using ResNet18-V.
• Lyric Word Error Rate (WER) (primary) — range: percent
  • Computed as (D+I+S)/N, where N is the number of words in the original source lyrics, and D, I, S are deletions, insertions, and substitutions in the transcribed output lyrics.
• Signal-to-Noise Ratio (SNR) — range: dB
  • 10log10(Px/Pδ), where Px is the power of the original singing voice and Pδ is the power of the perturbation.
• Perceptual Evaluation of Speech Quality (PESQ) — range: [-0.5, 4.5]
  • Objective perceptual metric simulating the human auditory system to measure audio quality and perturbation imperceptibility.

Input / output format

Input: Source singing voice, target singing voice(s), and a backing track (cropped 'Amazing Grace' to match length).

Output: Converted singing voice audio generated by the SVC model, along with the adversarially perturbed input audio files.

Scoring recipe

def evaluate_songbsab(output_audio, source_audio, target_audio, backing_track):
    # 1. Identity Similarity
    target_feat = extract_identity_feature(target_audio, model='ResNet18-V')
    output_feat = extract_identity_feature(output_audio, model='ResNet18-V')
    identity_sim = cosine_similarity(target_feat, output_feat)

    # 2. Lyric WER
    source_lyrics = get_ground_truth_lyrics(source_audio)
    output_lyrics = transcribe_lyrics(output_audio, model='Conformer')
    wer = (edit_distance(source_lyrics, output_lyrics) / len(source_lyrics)) * 100

    # 3. SNR & PESQ (merge stereo to mono first)
    mono_out = merge_stereo_to_mono(output_audio, backing_track)
    mono_src = merge_stereo_to_mono(source_audio, backing_track)
    perturbation = mono_out - mono_src
    snr = 10 * log10(power(mono_src) / power(perturbation))
    pesq = compute_pesq(mono_src, mono_out)
    return identity_sim, wer, snr, pesq

Common pitfalls

• SNR and PESQ must be computed on mono audio after merging the singing voice and backing track, not on the stereo channels separately.
• Identity similarity is evaluated using an external ResNet18-V encoder, not the identity encoder used inside the SVC models.
• The evaluation dataset is not a random split; it consists of the top 1,000 pairs with the highest initial identity similarity to test protection on the hardest conversion cases.

Evidence (verbatim from paper)

The following objective metrics will be used to evaluate SongBsAb. Cosine similarity between the centroid identity feature of the target singer and the identity feature of the output singing voice is used to measure identity disruption. We use the Resnet18 for verification (Res18-V) as the speaker recognition model for extracting identity features, which differs from the identity encoders used in SVC models and transferability analysis. Lyric word error rate (WER) of the output singing voice w.r.t. its original input source singing voice is used to measure lyric disruption. WER is computed as: WER=(D+I+S)/N where N is the number of words in the original source singing voice and D, I, S are the number of deletions, insertions, and substitutions in the output singing voice, respectively. To recognize the lyrics of a singing voice, we use the speech recognition model, Conformer, trained on the Chinese speech dataset WenetSpeech for Opensinger and the English speech dataset GigaSpeech for NUS-48E. Signal-to-Noise Ratio (SNR) and Perceptual Evaluation of Speech Quality (PESQ) are used to measure the imperceptibility of perturbations and the utility of SongBsAb. SNR is defined as 10log10

Citation

@misc{chen2024songbsab,
  title={SongBsAb: A Dual Prevention Approach against Singing Voice Conversion based Illegal Song Covers},
  author={Guangke Chen et al. (2024)},
  year={2024},
  note={arXiv:2401.17133}
}

• arXiv: 2401.17133