# Tifa 100 Eval

> Evaluates how well an automatic multimodal transformer predicts fine-grained human feedback (quality scores, region-level misalignment heatmaps, and text misalignment annotations) on text-to-image generation tasks. Use when the user wants to benchmark on TIFA, or asks about evaluating this task. Reports correlation.

- Skill: `qhjqhj00/tifa-100-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/tifa-100-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/tifa-100-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/tifa-100-eval

---


# tifa-100-eval

> Rich Human Feedback for Text-to-Image Generation — Liang et al. (2023) (arXiv:2312.10240, 2023)

## What this evaluates

Evaluates how well an automatic multimodal transformer predicts fine-grained human feedback (quality scores, region-level misalignment heatmaps, and text misalignment annotations) on text-to-image generation tasks.

## Datasets

- **TIFA** — total 100; splits: test (100)

## Metrics

- `correlation` **(primary)** — range: [-1, 1]
  - Measures the statistical correlation between the model's predicted scores, heatmaps, and misalignment annotations and the corresponding human-annotated ground truth labels.

## Input / output format

**Input**: A text prompt and its corresponding generated image.

**Output**: A scalar quality score, a region-level implausibility/misalignment heatmap, and a list of misaligned prompt keywords.

## Scoring recipe

```python
def compute_correlation(predictions, human_labels):
    # predictions and human_labels are arrays of floats
    # Compute Pearson correlation coefficient
    corr = np.corrcoef(predictions, human_labels)[0, 1]
    return corr
```

## Common pitfalls

- Evaluation is strictly limited to 100 prompts due to the high cost of human annotation.
- Relies entirely on human feedback as ground truth, which may introduce annotator bias or variability.

## Evidence (verbatim from paper)

> For the 100 prompts for our human evaluation, they are sampled from the existing benchmark: TIFA [24]. We only did our evaluation on 100 prompts due to the high cost of the human annotation. The model predicts these annotations with high correlation to human labels.

## Citation

```bibtex
@misc{liang2023richhumanfeedback,
  title={Rich Human Feedback for Text-to-Image Generation},
  author={Liang et al. (2023)},
  year={2023},
  note={arXiv:2312.10240}
}
```

- arXiv: 2312.10240