Scenefake Eval

scenefake-eval

SceneFake: An Initial Dataset and Benchmarks for Scene Fake Audio Detection — Jiangyan Yi et al. (2022) (arXiv:2211.06073, 2022)

What this evaluates

This benchmark evaluates audio forensics models on their ability to discriminate between genuine recordings and audio manipulated via acoustic scene forgery using speech enhancement technologies. It specifically measures threshold-free equal error rate (EER) to assess how well models generalize to unseen attacks without relying on a fixed decision boundary.

Datasets

• SceneFake — total ?; splits: test (-1); repo https://github.com/ADDchallenge/SceneFake

Metrics

• EER (primary) — range: [0, 1]
  • Equal Error Rate (EER) is the point where the false alarm rate P_fa and miss rate P_miss are equal. It is computed as a threshold-free metric: EER = P_fa(theta_EER) = P_miss(theta_EER), where theta_EER is the threshold that minimizes the difference between the two rates. Lower EER indicates better detection performance.

Input / output format

Input: Audio trials (either genuine or manipulated) provided to a detection algorithm, which outputs a continuous real-valued score.

Output: A single real-valued detection score per audio trial.

Scoring recipe

def compute_threshold_free_eer(genuine_scores, manipulated_scores):
    all_scores = genuine_scores + manipulated_scores
    thresholds = np.linspace(min(all_scores), max(all_scores), 1000)
    min_diff = float('inf')
    best_theta = thresholds[0]
    for theta in thresholds:
        pfa = np.mean(manipulated_scores > theta)
        pmiss = np.mean(genuine_scores < theta)
        diff = abs(pfa - pmiss)
        if diff < min_diff:
            min_diff = diff
            best_theta = theta
    eer = np.mean([np.mean(manipulated_scores > best_theta), np.mean(genuine_scores < best_theta)])
    return eer

Common pitfalls

• The metric is explicitly threshold-free; do not apply a fixed decision threshold or optimize one on the test set.
• Score direction is inverted compared to some benchmarks: high scores indicate genuine audio, low scores indicate manipulated audio.
• Lower EER values indicate better model performance, not higher.

Evidence (verbatim from paper)

The goal of audio scene manipulation detection is to develop a method or an algorithm to discriminate between the manipulated audio and the genuine one. So equal error rate (EER) is used as the evaluation metric for the detection tasks. ... The metric in this paper is the ’threshold-free’ EER, defined as follows. Let $P_{fa}(\theta)$ and $P_{miss}(\theta)$ denote the false alarm and miss rates at threshold $\theta$. ... So $P_{fa}(\theta)$ and $P_{miss}(\theta)$ are, respectively, monotonically decreasing and increasing functions of $\theta$. The EER corresponds to the threshold $\theta_{EER}$ at which the two detection error rates are equal, i.e. $EER=P_{fa}(\theta_{EER})=P_{miss}(\theta_{EER})$. It is the lower value of EER, the better performance of the model.

Citation

@misc{yi2022scenefake,
  title={SceneFake: An Initial Dataset and Benchmarks for Scene Fake Audio Detection},
  author={Jiangyan Yi et al. (2022)},
  year={2022},
  note={arXiv:2211.06073}
}

• arXiv: 2211.06073