Osbench Eval

osbench-eval

OpenSubject: Leveraging Video-Derived Identity and Diversity Priors for Subject-driven Image Generation and Manipulation — Liu et al. (2025) (arXiv:2512.08294, 2025)

What this evaluates

Evaluates subject-driven image generation and manipulation capabilities, specifically testing identity consistency, prompt adherence, and background preservation across single- and multi-subject scenarios.

Datasets

• OSBench — total 240; splits: test (240); repo https://github.com/LAW1223/OpenSubject

Metrics

• Prompt Adherence (PA) — range: [0, 10]
  • VLM judge (GPT-4.1) assigns a 0–10 score based on attribute, count, and relation compliance with the prompt.
• Identity Fidelity (IF) — range: [0, 10]
  • VLM judge assigns a 0–10 score measuring consistency with the subject across provided reference images.
• Overall (Generation) (primary) — range: [0, 10]
  • Geometric mean of PA and IF: sqrt(PA * IF).
• Manipulation Fidelity (MF) — range: [0, 10]
  • VLM judge assigns a 0–10 score measuring match between edited regions and referenced subject(s).
• Background Consistency (BC) — range: [0, 10]
  • VLM judge assigns a 0–10 score measuring stability of non-edited regions.
• Overall (Manipulation) — range: [0, 10]
  • Geometric mean of MF and BC: sqrt(MF * BC).

Input / output format

Input: Reference image(s) and an open-ended text prompt (generation); reference image(s), target scene image, and edit instruction (manipulation).

Output: A single synthesized or edited image.

Scoring recipe

judge = GPT4.1()
scores = {}
for task in [single_gen, multi_gen, single_manip, multi_manip]:
    for sample in task.samples:
        output = model.generate(sample.ref_images, sample.prompt)
        scores[task] = judge.score(output, sample, rubric=task.rubric, scale=10)
gen_overall = (scores['PA'] * scores['IF']) ** 0.5
manip_overall = (scores['MF'] * scores['BC']) ** 0.5
return gen_overall, manip_overall

Common pitfalls

• VLM judges may conflate aesthetic quality with strict prompt adherence or identity fidelity.
• Geometric mean heavily penalizes models that score high on one dimension but low on another, potentially masking balanced performance.
• Rubricized prompts for GPT-4.1 are not publicly detailed, making exact score reproduction difficult.

Evidence (verbatim from paper)

Following instruction-based assessment methods (e.g., VIEScore, OmniContext), we use a strong VLM judge (GPT-4.1) to assign 0–10 scores with rubricized prompts and independent criteria. For generation tasks, we report Prompt Adherence (PA) (attribute/count/relation compliance), Identity Fidelity (IF) (consistency with the subject across provided references), and Overall (geometric mean of PA and IF). For manipulation tasks, we report Manipulation Fidelity (MF) (match between edited regions and the referenced subject(s)), Background Consistency (BC) (stability of non-edited regions), and Overall (geometric mean of MF and BC).

Citation

@misc{liu2025opensubject,
  title={OpenSubject: Leveraging Video-Derived Identity and Diversity Priors for Subject-driven Image Generation and Manipulation},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2512.08294}
}

• arXiv: 2512.08294