Audio Compositionality Eval

audio-compositionality-eval

Evaluating Compositional Structure in Audio Representations — Chen et al. (2026) (arXiv:2603.13685, 2026)

What this evaluates

Probes whether audio encoders preserve algebraic consistency when identical sources are added to different base scenes (A-COAT), and whether representations can be accurately reconstructed from discrete attribute-level primitives like timbre, pitch, rate, and amplitude (A-TRE).

Datasets

• Synthetic Audio Scenes — total ?; splits: test (-1); repo https://github.com/chuyangchencd/audio-compositionality

Metrics

• A-COAT (primary) — range: [-1, 1]
  • Cosine similarity between additive transformation difference vectors: $\frac{\langle z_{B}-z_{A}, z_{D}-z_{C}\rangle}{|z_{B}-z_{A}||z_{D}-z_{C}|}$. Measures alignment of how the encoder represents adding the same sources to different base scenes.
• A-TRE — range: [-1, 1]
  • Cosine similarity between the encoder's scene embedding and a prediction made from attribute-level token vectors aggregated via a Transformer: $\frac{\langle z, \hat{z}\rangle}{|z||\hat{z}|}$. Measures reconstructibility from compositional primitives.

Input / output format

Input: A-COAT: quadruple of synthetic audio scenes $(A,B,C,D)$ where $B=A\cup T$ and $D=C\cup T$. A-TRE: single synthetic audio scene $X$ with discrete attribute metadata (timbre, pitch, rate, amplitude per source).

Output: Embedding vector $z_X \in \mathbb{R}^D$ produced by the audio encoder for each scene.

Scoring recipe

import numpy as np
def compute_acoat(z_A, z_B, z_C, z_D):
    d1 = z_B - z_A
    d2 = z_D - z_C
    return np.dot(d1, d2) / (np.linalg.norm(d1) * np.linalg.norm(d2))
def compute_atre(z, z_hat):
    return np.dot(z, z_hat) / (np.linalg.norm(z) * np.linalg.norm(z_hat))

Common pitfalls

• A-COAT requires computing cosine similarity on difference vectors, not raw scene embeddings; direct similarity will yield incorrect alignment scores.
• A-TRE involves training a lightweight composition model $g_\theta$ on a training split before evaluation; the metric measures alignment of the predicted embedding $\hat{z}$ with ground truth $z$, not the encoder's raw output alone.
• Synthetic scenes use discrete attribute classes; inconsistent discretization or source ordering across scenes can artificially inflate or deflate A-TRE reconstruction scores.

Evidence (verbatim from paper)

For an encoder $f$ producing embeddings $z_{X}=f(X)$, we evaluate whether the difference vectors $z_{B}-z_{A}$ and $z_{D}-z_{C}$ align. The A-COAT score is their cosine similarity: $\mathrm{A\text{-}COAT}(A,B,C,D)=\frac{\langle z_{B}-z_{A},;z_{D}-z_{C}\rangle}{|z_{B}-z_{A}|,|z_{D}-z_{C}|}.$ The score lies in $[-1,1]$, where $1$ indicates perfect alignment... The A-TRE score for a scene is the cosine similarity between the encoder and predicted embeddings: $\mathrm{A\text{-}TRE}(X)=\frac{\langle z,\hat{z}\rangle}{|z|;|\hat{z}|},\qquad z=f(X).$

Citation

@misc{chen2026evaluatingcompositional,
  title={Evaluating Compositional Structure in Audio Representations},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2603.13685}
}

• arXiv: 2603.13685