Streammark Eval

streammark-eval

StreamMark: A Deep Learning-Based Semi-Fragile Audio Watermarking for Proactive Deepfake Detection — Liu et al. (2026) (arXiv:2604.11917, 2026)

What this evaluates

This benchmark evaluates the imperceptibility, robustness to benign audio transformations, and semi-fragility to malicious deepfake manipulations of a deep learning-based audio watermarking system. It specifically probes whether a watermark can survive standard compression and cropping while being deliberately destroyed by semantic-altering AI conversions like voice cloning or speech editing.

Datasets

• LibriSpeech (train_clean100) — total ?; splits: train (-1)
• Test Set A — total ?; splits: test_A (-1)
• Test Set B — total ?; splits: test_B (-1); repo https://github.com/L1uZhentao/deepfake_benchmark

Metrics

• SNR — range: other
  • Signal-to-Noise Ratio in decibels, measuring the power ratio between the clean audio and the watermarked audio.
• PESQ — range: other
  • Perceptual Evaluation of Speech Quality, a standardized objective measure of speech quality that correlates with human perception.
• SECS — range: [0, 1]
  • Speaker Encoder Cosine Similarity, computed as the cosine similarity between speaker embeddings extracted from the clean and watermarked audio.
• ACC (primary) — range: percent
  • Message Recovery Accuracy, calculated as the percentage of correctly recovered bits out of the total 16-bit message length.

Input / output format

Input: Clean audio waveform (from LibriSpeech test_clean subset)

Output: Watermarked audio waveform (for imperceptibility/robustness evaluation) and/or decoded 16-bit binary message (for ACC evaluation)

Scoring recipe

def score(predictions, gold):
    clean_audio, original_msg = gold
    watermarked_audio, decoded_msg = predictions
    snr = calculate_snr(clean_audio, watermarked_audio)
    pesq = calculate_pesq(clean_audio, watermarked_audio)
    secs = cosine_similarity(speaker_embed(clean_audio), speaker_embed(watermarked_audio))
    acc = sum(p == g for p, g in zip(decoded_msg, original_msg)) / len(original_msg) * 100
    return {'SNR': snr, 'PESQ': pesq, 'SECS': secs, 'ACC': acc}

Common pitfalls

• Message Recovery Accuracy (ACC) dropping to ~50% under deepfake attacks is the intended 'fragile' behavior (equivalent to random guessing for 16-bit messages), not a model failure.
• The benchmark explicitly distinguishes between benign (semantic-preserving) and malicious (semantic-altering) AI conversions; misclassifying a benign style transfer as a deepfake attack violates the semi-fragility design goal.
• Opus encoding is an out-of-distribution attack not seen during training; high robustness here may not generalize to other codecs without retraining.

Evidence (verbatim from paper)

The primary evaluation metrics were Signal-to-Noise Ratio (SNR), Perceptual Evaluation of Speech Quality (PESQ) , and Speaker Encoder Cosine Similarity (SECS) for imperceptibility, and Message Recovery Accuracy (ACC) for robustness and fragility.

Citation

@misc{liu2026streammark,
  title={StreamMark: A Deep Learning-Based Semi-Fragile Audio Watermarking for Proactive Deepfake Detection},
  author={Liu et al. (2026)},
  year={2026},
  note={arXiv:2604.11917}
}

• arXiv: 2604.11917