Artifactbench Eval

artifactbench-eval

ArtifactNet: Detecting AI-Generated Music via Forensic Residual Physics — Oh et al. (2026) (arXiv:2604.16254, 2026)

What this evaluates

Evaluates the ability to detect AI-generated music by identifying irreversible residual artifacts from neural audio codecs. It probes robustness across diverse generators, lossy compression codecs, and adversarial source-separation attacks, while measuring false-positive rates on real-world music.

Datasets

• ArtifactBench v1 — total 6183; splits: test (2263); HF intrect/artifactbench-v1

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: F1 = 2 * (Precision * Recall) / (Precision + Recall).
• FPR — range: [0, 1]
  • False Positive Rate: proportion of real music incorrectly classified as AI-generated. FPR = FP / (FP + TN).
• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve, measuring discriminative ability across all classification thresholds.

Input / output format

Input: Raw audio tracks (WAV, MP3, AAC, Opus) sampled at 44.1 kHz.

Output: Binary classification probability P(AI) or discrete label (AI/Real) based on a decision threshold.

Scoring recipe

def compute_metrics(predictions, labels, threshold=0.5):
    preds_binary = (predictions >= threshold).astype(int)
    tp = np.sum((preds_binary == 1) & (labels == 1))
    fp = np.sum((preds_binary == 1) & (labels == 0))
    fn = np.sum((preds_binary == 0) & (labels == 1))
    tn = np.sum((preds_binary == 0) & (labels == 0))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    fpr = fp / (fp + tn) if (fp + tn) > 0 else 0
    tpr = recall
    auc = roc_auc_score(labels, predictions)
    return {'F1': f1, 'FPR': fpr, 'AUC': auc}

Common pitfalls

• Confusing lossy compression artifacts (e.g., MP3/AAC) with neural codec RVQ artifacts, leading to high false positives on real music.
• Threshold sensitivity: performance varies significantly if the decision threshold deviates from the standard 0.5.
• Distribution shift: models trained on specific generators fail to generalize to unseen AI music generators or diverse real-music production styles.

Evidence (verbatim from paper)

All three models are evaluated on ArtifactBench — a continuously-updated benchmark we constructed for this work — under identical conditions: same audio files, same preprocessing, threshold $ au = 0.5$ (consistent with the authors' inference code). For fair zero-shot comparison, we restrict evaluation to the test partition (bench_origin=test, n=2,263), unseen by all three models during training. ArtifactNet achieves the highest performance across all metrics (F1 = 0.9829 vs. 0.7576 for CLAM and 0.7713 for SpecTTTra), while using $49 imes$ fewer parameters than CLAM and $4.8 imes$ fewer than SpecTTTra.

Citation

@misc{oh2026artifactnet,
  title={ArtifactNet: Detecting AI-Generated Music via Forensic Residual Physics},
  author={Oh et al. (2026)},
  year={2026},
  note={arXiv:2604.16254}
}

• arXiv: 2604.16254