Sketch Less Retrieval Eval

sketch-less-retrieval-eval

15M Multimodal Facial Image-Text Dataset — Dawei Dai et al. (2024) (arXiv:2407.08515, 2024)

What this evaluates

Evaluates early retrieval performance using partial or low-quality sketches combined with text, measuring how quickly the correct target image appears in the ranked list as the sketch is drawn. It probes robustness to incomplete visual inputs and multimodal fusion.

Datasets

• FS2K-SDE1 — total ?; splits: train (76609), test (-1)
• FS2K-SDE2 — total ?; splits: train (23714), test (-1)
• User-SDE — total ?; splits: test (110)

Metrics

• m@A, m@B (primary) — range: percent
  • m@A is the ranking percentile (higher indicates better early retrieval), and m@B is 1/rank versus percentage of sketch completion. Both capture retrieval efficiency as the sketch is progressively drawn.

Input / output format

Input: A partial sketch of a face and its corresponding text description.

Output: A ranked list of target face images.

Scoring recipe

def compute_slfir(predictions, gold, sketch_pct):
    ranks = [pred_list.index(g) + 1 for pred_list, g in zip(predictions, gold)]
    m_a = np.mean([100 - (r / len(predictions) * 100) for r in ranks])
    m_b = np.mean([1.0 / r for r in ranks]) * 100
    return m_a, m_b

Common pitfalls

• Metrics are task-specific to early retrieval; m@A measures ranking percentile while m@B measures retrieval efficiency relative to sketch completion percentage, requiring careful interpretation.
• User-SDE tests generalization across different painters, which typically shows performance drops compared to controlled datasets due to sketching diversity.

Evidence (verbatim from paper)

m@A (the ranking percentile) and m@B (1/rank versus percentage of sketch) were used to capture the retrieval performance for the partial sketches.

Citation

@misc{dai2024facecaption,
  title={15M Multimodal Facial Image-Text Dataset},
  author={Dawei Dai et al. (2024)},
  year={2024},
  note={arXiv:2407.08515}
}

• arXiv: 2407.08515