Symile Synthetic Eval

symile-synthetic-eval

Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities — Saporta et al. (2024) (arXiv:2411.01053, 2024)

What this evaluates

Probes a model's ability to capture higher-order conditional dependencies between modalities by predicting one modality's representation from two others under varying information dynamics. It specifically tests whether a model can leverage joint information when pairwise mutual information is zero.

Datasets

• Synthetic dataset — total ?; splits: test (-1); repo https://github.com/rajesh-lab/symile

Metrics

• mean accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted target representations given input pairs. Reported as the mean across 10 bootstrap samples of the test set.

Input / output format

Input: Pair of learned representations $(\mathbf{r}{\mathbf{a}}, \mathbf{r}{\mathbf{c}})$ derived from affine linear functions mapping input vectors $\mathbf{a}, \mathbf{c} \in \mathbb{R}^5$.

Output: Predicted representation vector $\mathbf{r}_{\mathbf{b}} \in \mathbb{R}^{16}$.

Scoring recipe

accuracy = 0
for each sample in test_set:
    pred = model.predict(r_a, r_c)
    if pred == r_b:
        accuracy += 1
return accuracy / len(test_set)

Common pitfalls

• Assuming pairwise mutual information is sufficient for prediction; the task explicitly relies on higher-order conditional dependencies that are zero pairwise.
• Not accounting for the bootstrap sampling procedure; results must be averaged over 10 test set resamples to match reported values.

Evidence (verbatim from paper)

Mean accuracy is reported across 10 bootstrap samples of the test set. Both models start with a mean accuracy of $0.032\pm 0.001$ (SE) at $\hat{p}=0$. As $\hat{p}$ increases, Symile’s accuracy progressively climbs, reaching perfect accuracy at $\hat{p}=1\pm 0.0$ (SE).

Citation

@misc{saporta2024symile,
  title={Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities},
  author={Saporta et al. (2024)},
  year={2024},
  note={arXiv:2411.01053}
}

• arXiv: 2411.01053