# Symile Synthetic Eval

> Probes a model's ability to capture higher-order conditional dependencies between modalities by predicting one modality's representation from two others under varying information dynamics. It specifically tests whether a model can leverage joint information when pairwise mutual information is zero. Use when the user wants to benchmark on Synthetic dataset, or asks about evaluating this task. Reports mean accuracy.

- Skill: `qhjqhj00/symile-synthetic-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/symile-synthetic-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/symile-synthetic-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/symile-synthetic-eval

---


# symile-synthetic-eval

> Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities — Saporta et al. (2024) (arXiv:2411.01053, 2024)

## What this evaluates

Probes a model's ability to capture higher-order conditional dependencies between modalities by predicting one modality's representation from two others under varying information dynamics. It specifically tests whether a model can leverage joint information when pairwise mutual information is zero.

## Datasets

- **Synthetic dataset** — total ?; splits: test (-1); repo https://github.com/rajesh-lab/symile

## Metrics

- `mean accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted target representations given input pairs. Reported as the mean across 10 bootstrap samples of the test set.

## Input / output format

**Input**: Pair of learned representations $(\mathbf{r}_{\mathbf{a}}, \mathbf{r}_{\mathbf{c}})$ derived from affine linear functions mapping input vectors $\mathbf{a}, \mathbf{c} \in \mathbb{R}^5$.

**Output**: Predicted representation vector $\mathbf{r}_{\mathbf{b}} \in \mathbb{R}^{16}$.

## Scoring recipe

```python
accuracy = 0
for each sample in test_set:
    pred = model.predict(r_a, r_c)
    if pred == r_b:
        accuracy += 1
return accuracy / len(test_set)
```

## Common pitfalls

- Assuming pairwise mutual information is sufficient for prediction; the task explicitly relies on higher-order conditional dependencies that are zero pairwise.
- Not accounting for the bootstrap sampling procedure; results must be averaged over 10 test set resamples to match reported values.

## Evidence (verbatim from paper)

> Mean accuracy is reported across 10 bootstrap samples of the test set. Both models start with a mean accuracy of $0.032\pm 0.001$ (SE) at $\hat{p}\=0$. As $\hat{p}$ increases, Symile’s accuracy progressively climbs, reaching perfect accuracy at $\hat{p}\=1\pm 0.0$ (SE).

## Citation

```bibtex
@misc{saporta2024symile,
  title={Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities},
  author={Saporta et al. (2024)},
  year={2024},
  note={arXiv:2411.01053}
}
```

- arXiv: 2411.01053