qhjqhj00 actionbench-eval
Learning Disentangled Identifiers for Action-Customized Text-to-Image Generation — Siteng Huang et al. (2023) (arXiv:2311.15841, 2023)
What this evaluates
Evaluates a text-to-image model's ability to customize generated images with a specific subject's appearance while accurately transferring a target action from an exemplar image, without appearance leakage or subject deformation.
Datasets
- ActionBench — total ?; splits: test (-1)
Metrics
action accuracy— range: percent- Percentage of generated images where the action matches the exemplar action, as judged by human evaluators.
subject accuracy— range: percent- Percentage of generated images where the subject matches the provided textual name without obvious deformations or defects.
total accuracy(primary) — range: percent- Percentage of generated images that are considered totally correct, meaning both the action and the subject are correctly generated according to human evaluators.
Input / output format
Input: Exemplar image(s) showing a specific action, and a textual name/description of the target subject.
Output: A single 512x512 RGB image generated by the text-to-image model.
Scoring recipe
correct_count = 0
total_count = 0
for pair in ActionBench:
images = model.generate(exemplar=pair.exemplar, subject_name=pair.name, n=4)
for img in images:
total_count += 1
action_ok = evaluator.check_action_consistency(img, pair.exemplar)
subject_ok = evaluator.check_subject_fidelity(img, pair.name)
if action_ok and subject_ok:
correct_count += 1
total_accuracy = (correct_count / total_count) * 100
Common pitfalls
- Relies entirely on human evaluation without reporting inter-annotator agreement or rater calibration.
- Only 4 images are sampled per subject-action pair, which may not capture generation variance or diversity.
- Binary 'totally correct' criterion ignores partial successes or graded quality assessments.
Evidence (verbatim from paper)
Tab. 1 reports the action, subject and total accuracy for all methods. ... A generated image will only be considered totally correct if both the action and the character are correctly generated.
Citation
@misc{huang2023learningdisentangledidentifiers,
title={Learning Disentangled Identifiers for Action-Customized Text-to-Image Generation},
author={Siteng Huang et al. (2023)},
year={2023},
note={arXiv:2311.15841}
}
- arXiv: 2311.15841