Nacsp Eval

nacsp-eval

Towards Neural Audio Codec Source Parsing — Phukan et al. (2025) (arXiv:2506.12627, 2025)

What this evaluates

Evaluates a model's ability to predict discrete neural audio codec parameters (quantizers, sampling rate, bits per second) from audio samples, enabling fine-grained source attribution of AI-generated speech. The protocol frames open-set attribution as a multi-task regression problem rather than binary classification, requiring the model to generalize across both seen and unseen codec configurations.

Datasets

• ST-Codecfake — total ?; splits: train (70000), val (7000), test (158736)
• CodecFake — total ?; splits: train (42752), val (735), test (755)

Metrics

• MSE (primary) — range: other
  • Mean Squared Error between predicted and ground truth values for quantizers (Q), sampling rate (SR), and bits per second (BPS).

Input / output format

Input: Raw audio waveform samples corresponding to speech synthesized by various neural audio codecs.

Output: Three predicted values per sample: quantizers (Q), sampling rate (SR), and bits per second (BPS).

Scoring recipe

def compute_mse(preds, golds):
    # preds, golds: (N, 3) arrays for [Q, SR, BPS]
    mse_q = np.mean((preds[:, 0] - golds[:, 0]) ** 2)
    mse_sr = np.mean((preds[:, 1] - golds[:, 1]) ** 2)
    mse_bps = np.mean((preds[:, 2] - golds[:, 2]) ** 2)
    return {'MSE_Q': mse_q, 'MSE_SR': mse_sr, 'MSE_BPS': mse_bps}

Common pitfalls

• The protocol explicitly restricts evaluation to fake samples only; including real audio violates the setup.
• The task is framed as multi-task regression for specific codec parameters, not binary real/fake classification.
• Dataset splits for CodecFake are provided per NAC subset; aggregating across subsets without stratification may skew results.

Evidence (verbatim from paper)

We use MSE (mean squared error) as loss function for all the three NACSP tasks (Q, BPS, SR) with Adam as the optimizer. We use only the fake samples and follow the distribution given by them for the training and evaluation of the models.

Citation

@misc{phukan2025towards,
  title={Towards Neural Audio Codec Source Parsing},
  author={Phukan et al. (2025)},
  year={2025},
  note={arXiv:2506.12627}
}

• arXiv: 2506.12627