Symile M3 Eval

symile-m3-eval

Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities — Saporta et al. (2024) (arXiv:2411.01053, 2024)

What this evaluates

Evaluates zero-shot cross-modal retrieval capability by requiring a model to jointly leverage audio and text to identify an image, where neither modality alone contains sufficient information. It tests the model's ability to capture joint information across three distinct high-dimensional data types.

Datasets

• Symile-M3 — total 33000000; splits: train (10000000), val (500000), test (500000); repo https://github.com/rajesh-lab/symile

Metrics

• mean accuracy (primary) — range: [0, 1]
  • Top-1 retrieval accuracy for predicting the correct image class given an audio-text query. Reported as mean across 10 bootstrap samples of the test set.

Input / output format

Input: Paired audio clip (one-sentence spoken in one of w languages) and text string (w words separated by underscores, containing the image class name in the audio's language).

Output: Predicted image class label from the 1,000 ImageNet classes.

Scoring recipe

candidates = test_set_image_classes
for each query in test_set:
    scores = model.compute_similarity(query.audio, query.text, candidates)
    pred = argmax(scores)
    if pred == query.true_image_class:
        correct += 1
return correct / len(test_set)

Common pitfalls

• Treating the task as pairwise retrieval; the dataset is explicitly constructed so that text or audio alone cannot predict the image, bounding pairwise models at 1/w accuracy.
• Ignoring the bootstrap resampling; the paper reports means over 10 bootstrap samples, not single-run accuracy.

Evidence (verbatim from paper)

We evaluate the learned representations on the zero-shot retrieval task of finding an image of the appropriate class given the audio and text. Symile successfully leverages this joint information, with mean accuracies of $0.939$, $0.919$, and $0.882$ on Symile-M3-2, Symile-M3-5, and Symile-M3-10, respectively, calculated across 10 bootstrap samples of the test set.

Citation

@misc{saporta2024symile,
  title={Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities},
  author={Saporta et al. (2024)},
  year={2024},
  note={arXiv:2411.01053}
}

• arXiv: 2411.01053