# Actionbench Eval

> Evaluates a text-to-image model's ability to customize generated images with a specific subject's appearance while accurately transferring a target action from an exemplar image, without appearance leakage or subject deformation. Use when the user wants to benchmark on ActionBench, or asks about evaluating this task. Reports total accuracy.

- Skill: `qhjqhj00/actionbench-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/actionbench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/actionbench-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/actionbench-eval

---


# actionbench-eval

> Learning Disentangled Identifiers for Action-Customized Text-to-Image Generation — Siteng Huang et al. (2023) (arXiv:2311.15841, 2023)

## What this evaluates

Evaluates a text-to-image model's ability to customize generated images with a specific subject's appearance while accurately transferring a target action from an exemplar image, without appearance leakage or subject deformation.

## Datasets

- **ActionBench** — total ?; splits: test (-1)

## Metrics

- `action accuracy` — range: percent
  - Percentage of generated images where the action matches the exemplar action, as judged by human evaluators.
- `subject accuracy` — range: percent
  - Percentage of generated images where the subject matches the provided textual name without obvious deformations or defects.
- `total accuracy` **(primary)** — range: percent
  - Percentage of generated images that are considered totally correct, meaning both the action and the subject are correctly generated according to human evaluators.

## Input / output format

**Input**: Exemplar image(s) showing a specific action, and a textual name/description of the target subject.

**Output**: A single 512x512 RGB image generated by the text-to-image model.

## Scoring recipe

```python
correct_count = 0
total_count = 0
for pair in ActionBench:
    images = model.generate(exemplar=pair.exemplar, subject_name=pair.name, n=4)
    for img in images:
        total_count += 1
        action_ok = evaluator.check_action_consistency(img, pair.exemplar)
        subject_ok = evaluator.check_subject_fidelity(img, pair.name)
        if action_ok and subject_ok:
            correct_count += 1
total_accuracy = (correct_count / total_count) * 100
```

## Common pitfalls

- Relies entirely on human evaluation without reporting inter-annotator agreement or rater calibration.
- Only 4 images are sampled per subject-action pair, which may not capture generation variance or diversity.
- Binary 'totally correct' criterion ignores partial successes or graded quality assessments.

## Evidence (verbatim from paper)

> Tab. 1 reports the action, subject and total accuracy for all methods. ... A generated image will only be considered totally correct if both the action and the character are correctly generated.

## Citation

```bibtex
@misc{huang2023learningdisentangledidentifiers,
  title={Learning Disentangled Identifiers for Action-Customized Text-to-Image Generation},
  author={Siteng Huang et al. (2023)},
  year={2023},
  note={arXiv:2311.15841}
}
```

- arXiv: 2311.15841