Memfeed Eval

memfeed-eval

How to Take a Memorable Picture? Empowering Users with Actionable Feedback — Laiti et al. (2026) (arXiv:2602.21877, 2026)

What this evaluates

Probes a model's ability to generate actionable, natural-language feedback to improve the memorability of a photograph at capture time. It evaluates both the effectiveness of the feedback in increasing image memorability and the linguistic coherence of the suggestions.

Datasets

• MemBench — total ?; splits: train (-1)

Metrics

• IR (primary) — range: [0, 1]
  • Proportion of instances where applying the generated feedback to the source image yields a higher memorability score than the original image.
• RM% — range: percent
  • Relative percentage increase in memorability score after applying the feedback-driven edit, calculated as ((score_after - score_before) / score_before) * 100.
• Perplexity — range: other
  • Language model perplexity over the ground-truth feedback text, measuring how closely the model's suggestions match human-written memorability-aware instructions.

Input / output format

Input: A source photograph, optionally paired with a reference destination image (for teacher oracles), and a prompt requesting actionable suggestions to improve the image's memorability.

Output: Natural-language textual instructions (e.g., 'emphasize facial expression', 'look at the camera') intended to guide image editing.

Scoring recipe

for each instance in dataset:
  feedback = model.generate(source_image)
  edited_image = apply_edit(source_image, feedback)
  mem_before = get_memorability(source_image)
  mem_after = get_memorability(edited_image)
  ir = 1 if mem_after > mem_before else 0
  rm_pct = ((mem_after - mem_before) / mem_before) * 100
  ppl = compute_perplexity(model, ground_truth_feedback)

IR = mean(ir)
RM% = mean(rm_pct)
Perplexity = mean(ppl)

Common pitfalls

• Confusing the teacher oracle setup (which uses privileged destination images) with the zero-shot or steered evaluation setup.
• Misinterpreting RM% as absolute memorability rather than relative percentage gain.
• Assuming lower perplexity directly correlates with higher memorability improvement, as they measure distinct linguistic and visual properties.

Evidence (verbatim from paper)

Table 2 reports the quantitative comparison of different MLLMs when asked for memorability feedback, as evaluated in terms of both editing metrics and perplexity. ... We observe a marked increase in IR, indicating that feedback produced by the steered model more frequently leads to edits that raise the memorability of the resulting images. This gain is further confirmed by a higher RM, showing that the relative increase in memorability is not only more frequent but also larger. ... Turning to the likelihood of ground-truth feedback, the lower perplexity achieved by MemCoach confirms its improved alignment with human-like memorability-aware feedback

Citation

@misc{laiti2026memcoach,
  title={How to Take a Memorable Picture? Empowering Users with Actionable Feedback},
  author={Laiti et al. (2026)},
  year={2026},
  note={arXiv:2602.21877}
}

• arXiv: 2602.21877