New York Smells Eval

new-york-smells-eval

New York Smells: A Large Multimodal Dataset for Olfaction — Ozguroglu et al. (2025) (arXiv:2511.20544, 2025)

What this evaluates

Evaluates multimodal representation learning for olfaction by testing cross-modal retrieval and classification tasks using paired image and electronic nose signals. Probes the model's ability to generalize from visual supervision to interpret raw or processed olfactory sensor data for scene, object, material, and fine-grained species recognition.

Datasets

• New York Smells — total 7000; splits: train (-1), test (-1)

Metrics

• classification accuracy (primary) — range: percent
  • Percentage of correctly predicted class labels out of the total number of samples. Computed separately for scene, material, and object categories, as well as fine-grained grass species discrimination.

Input / output format

Input: Paired olfactory sensor data (either raw $T \times 32$ resistance matrix or 32-dimensional Smellprint vector) and corresponding RGB images. For evaluation tasks, the model receives only the olfactory input to predict visual categories or retrieve matching images.

Output: Class labels (e.g., scene, material, object, or grass species) for classification tasks; ranked list of images for cross-modal retrieval.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

Common pitfalls

• The paper evaluates both end-to-end learning (Scratch) and linear probing on frozen self-supervised features (SSL), which yield different performance profiles and should not be conflated.
• The hand-crafted 'Smellprint' feature discards second-order statistics and temporal dynamics present in the raw sensor matrix, leading to significant performance drops compared to raw signal baselines.
• The contrastive pretraining uses a fixed temperature $\tau=0.07$ and symmetric loss; changing this or using asymmetric retrieval metrics without explicit definition may misalign with the reported benchmarks.

Evidence (verbatim from paper)

We evaluate classification accuracy at recognizing scenes, objects, and materials from smell alone. For each approach, we compare end-to-end learning with scratch initialization (Scratch), self-supervised representations with linear probes (SSL), and linear probe with random weights (Rand).

Citation

@misc{ozguroglu2025newyorksmells,
  title={New York Smells: A Large Multimodal Dataset for Olfaction},
  author={Ozguroglu et al. (2025)},
  year={2025},
  note={arXiv:2511.20544}
}

• arXiv: 2511.20544