Nsynth Reconstruction Eval

nsynth-reconstruction-eval

Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders — Engel et al. (2017) (arXiv:1704.01279, 2017)

What this evaluates

Evaluates the ability of audio generative models to reconstruct raw musical note waveforms and interpolate timbre and pitch in a learned latent space. Probes phase preservation, harmonic structure modeling, and the decoupling of pitch and timbre information.

Datasets

• NSynth — total 300000; splits: test (-1)

Metrics

• Classification accuracy (primary) — range: percent
  • Percentage of correctly predicted pitch or quality labels by a trained classifier on model outputs or latent embeddings.

Input / output format

Input: Raw audio waveforms of musical notes (mu-law encoded), latent embeddings (Z), and pitch conditioning variables.

Output: Reconstructed audio waveforms, interpolated audio samples, and latent embeddings.

Scoring recipe

clf = train_classifier(labels=["pitch", "quality"], data=reconstructions)
preds = clf.predict(held_out_reconstructions[:4096])
accuracy = (preds == gold_labels).mean() * 100

Common pitfalls

• Audio quality heavily depends on phase information, which standard magnitude spectrograms fail to capture, leading to misleading visual evaluations.
• The quantitative classifier is evaluated on a fixed subset of 4096 examples, which may not represent the full distribution of the ~300k dataset.
• Mu-law encoding introduces built-in distortion that affects lower frequencies more severely, confounding pure reconstruction quality metrics.

Evidence (verbatim from paper)

Table 1: Classification accuracy of a deep nonlinear pitch and quality classifier on reconstructions of a test set.

Citation

@misc{engel2017neural,
  title={Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders},
  author={Engel et al. (2017)},
  year={2017},
  note={arXiv:1704.01279}
}

• arXiv: 1704.01279