Sytone Disentanglement Eval

sytone-disentanglement-eval

Learning Disentangled Audio Representations through Controlled Synthesis — Brima et al. (2024) (arXiv:2402.10547, 2024)

What this evaluates

Evaluates the ability of representation learning models to factorize audio into independent semantic factors (timbre, amplitude, frequency). It measures how well the learned latent space aligns with these ground-truth factors using standard disentanglement metrics.

Datasets

• SynTone — total ?; splits: train (-1), test (-1)

Metrics

• MIG (primary) — range: [0, 1]
  • Mutual Information Gap. Measures the difference between the maximum and average mutual information between a single latent dimension and all ground-truth factors.
• SAP — range: [0, 1]
  • Separate Accuracy. Computes the fraction of ground-truth factors that can be predicted from a single latent dimension with accuracy significantly above chance.
• DCI — range: [0, 1]
  • Disentanglement, Completeness, Isolation. Product of completeness (how well factors are captured) and isolation (how concentrated factor information is in specific dimensions).
• JEMMIG — range: [0, 1]
  • Joint Entropy Mutual Information Gap. Extends MIG by considering joint entropy of latent dimensions to measure disentanglement.
• Mod. Score — range: [0, 1]
  • Measures how well the latent dimensions can be partitioned to predict each ground-truth factor, penalizing cross-factor predictions.

Input / output format

Input: 2D time-frequency representation of audio clips with known ground-truth factors (timbre, amplitude, frequency).

Output: Latent vector representations from the VAE encoder, evaluated against ground-truth factors to compute disentanglement scores.

Scoring recipe

def compute_disentanglement(latents, factors, n_runs=10):
    scores = {}
    for run in range(n_runs):
        # Estimate mutual information I(z_i; f_j) using k-NN or classifier
        mi_matrix = estimate_mutual_information(latents, factors)
        # MIG: max_j I(z_i; f_j) - mean_j I(z_i; f_j) averaged over i
        scores['MIG'] = compute_mig(mi_matrix)
        # SAP: fraction of factors predictable from single z_i
        scores['SAP'] = compute_separate_accuracy(latents, factors)
        # DCI: completeness * isolation
        scores['DCI'] = compute_dci(mi_matrix)
        # JEMMIG & Modularity Score follow similar MI-based formulations
    return mean(scores), std(scores)

Common pitfalls

• Metrics are highly sensitive to the number of samples and the choice of estimator for mutual information.
• Higher compactness scores (MIG, SAP) do not guarantee semantically meaningful factors; they may simply reflect overfitting to dataset structure.
• Standard deviations in the paper are reported as ±0.0, which likely indicates rounding or insufficient evaluation runs to capture variance.

Evidence (verbatim from paper)

Table 1 provides a summary of these metrics for each model. ... Disentanglement metrics with standard deviation for different models over 10 evaluation runs each. In our analysis, the vanilla VAE’s superior performance in compactness metrics (MIG, SAP) and comparable DCIMIG to β-TCVAE was unexpected, especially as compactness is of lesser practical importance.

Citation

@misc{brima2024learningdisentangled,
  title={Learning Disentangled Audio Representations through Controlled Synthesis},
  author={Brima et al. (2024)},
  year={2024},
  note={arXiv:2402.10547}
}

• arXiv: 2402.10547