Textme Eval

textme-eval

TextME: Bridging Unseen Modalities Through Text Descriptions — Soyeon Hong et al. (arXiv:2602.03098, 2026)

What this evaluates

Evaluates zero-shot cross-modal retrieval and classification across six modalities (image, video, audio, 3D, X-ray, molecules) using a text-only expansion framework. It probes whether unpaired text descriptions can bridge the geometric modality gap to align diverse modalities into a unified LLM embedding space without paired supervision.

Datasets

• COCO — total ?; splits: test (-1)
• Flickr30k — total ?; splits: test (-1)
• MSR-VTT — total ?; splits: test (-1)
• MSVD — total ?; splits: test (-1)
• DiDeMo — total ?; splits: test (-1)
• AudioCaps — total ?; splits: test (-1)
• Clotho — total ?; splits: test (-1)
• DrugBank — total ?; splits: test (-1)
• AudioSet — total ?; splits: test (-1)
• ESC-50 — total ?; splits: test (-1)
• ModelNet40 — total ?; splits: test (-1)
• ScanObjectNN — total ?; splits: test (-1)
• RSNA — total ?; splits: test (-1)

Metrics

• Recall@k (R@k) (primary) — range: percent
  • Percentage of queries where the ground-truth item appears in the top-k retrieved results based on cosine similarity in the shared embedding space.
• MRR@k — range: percent
  • Mean Reciprocal Rank for molecule retrieval, averaging the inverse rank of the first correct match across queries.
• Top-k accuracy — range: percent
  • Percentage of correctly classified instances where the predicted label matches the ground truth within the top-k predictions.
• Performance Preservation Ratio (PPR) — range: percent
  • Relative metric calculated as (TextME score / Pretrained encoder score) × 100%, measuring how much of the original encoder's capability is retained after text-only expansion.

Input / output format

Input: Text descriptions and unpaired modality instances (images, videos, audio, 3D models, X-rays, molecules) independently projected into a shared LLM embedding space.

Output: Ranked list of retrieved items for retrieval tasks, or predicted class labels for classification tasks.

Scoring recipe

def compute_recall_at_k(sim_matrix, k):
    ranks = np.argsort(-sim_matrix, axis=1)
    correct = np.any(ranks[:, :k] == np.arange(sim_matrix.shape[0]), axis=1)
    return np.mean(correct) * 100

def compute_mrr(sim_matrix):
    ranks = np.argsort(-sim_matrix, axis=1)
    ranks_correct = np.where(ranks == np.arange(sim_matrix.shape[0])[:, None])[1]
    return np.mean(1.0 / (ranks_correct + 1)) * 100

def compute_topk_accuracy(preds, labels, k=1):
    correct = np.sum(np.isin(preds, labels))
    return (correct / len(labels)) * 100

def compute_ppr(method_score, pretrained_score):
    return (method_score / pretrained_score) * 100

Common pitfalls

• Assuming paired modality-text data is required for training; the protocol explicitly uses only unpaired text descriptions (100K per modality).
• Interpreting PPR as absolute performance rather than a relative measure of preserved pretrained encoder capability.
• Applying offset correction blindly without verifying gap consistency, which can degrade performance for modalities like molecules.

Evidence (verbatim from paper)

We report Recall@k (R@k) for retrieval, MRR@k for molecule retrieval following Liu et al. (2023), and Top-k accuracy for classification. We define Performance Preservation Ratio (PPR) as the percentage of pretrained encoder performance retained by our method: PPR = (TextME score / Pretrained score) × 100%.

Citation

@misc{hong2026textme,
  title={TextME: Bridging Unseen Modalities Through Text Descriptions},
  author={Soyeon Hong et al.},
  year={2026},
  note={arXiv:2602.03098}
}

• arXiv: 2602.03098