Recipe1mplus Retrieval Eval

recipe1mplus-retrieval-eval

Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images — Marín et al. (2018) (arXiv:1810.06553, 2018)

What this evaluates

Evaluates cross-modal retrieval between food images and cooking recipes. It probes a model's ability to align visual and textual representations in a shared embedding space to rank relevant recipes given an image, and vice versa.

Datasets

• Recipe1M+ — total ?; splits: test (1000)

Metrics

• medR (primary) — range: other
  • Median rank of the ground-truth item in the retrieved list. Lower values indicate better retrieval performance.
• R@K — range: percent
  • Recall at top K: the percentage of queries where the ground-truth item appears in the top K retrieved results. Higher values indicate better performance.

Input / output format

Input: A food image (for im2recipe retrieval) or a cooking recipe text (for recipe2im retrieval).

Output: A ranked list of candidate recipes (for im2recipe) or images (for recipe2im) from the test set, ordered by cosine similarity in the shared embedding space.

Scoring recipe

def compute_metrics(predictions, gold):
    ranks = []
    for pred, gt in zip(predictions, gold):
        rank = pred.index(gt) + 1
        ranks.append(rank)
    medR = np.median(ranks)
    r_at_k = {k: np.mean([1 if r <= k else 0 for r in ranks]) for k in [1, 5, 10]}
    return medR, r_at_k

Common pitfalls

• MedR is lower-is-better, while R@K is higher-is-better; confusing the direction leads to misinterpreting results.
• The official evaluation uses a fixed subset of 1,000 test pairs repeated 10 times, not the full dataset.
• Human baseline uses a 10-way multiple-choice setup, which is not directly comparable to open retrieval without adjustment.

Evidence (verbatim from paper)

We report median rank (MedR), and recall rate at top $K$ (R@K) for all the retrieval experiments. To clarify, R@5 in the im2recipe task represents the percentage of all the image queries where the corresponding recipe is retrieved in the top $5$, hence higher is better.

Citation

@misc{marin2018recipe1mplus,
  title={Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images},
  author={Marín et al. (2018)},
  year={2018},
  note={arXiv:1810.06553}
}

• arXiv: 1810.06553