Tifa 100 Eval

tifa-100-eval

Rich Human Feedback for Text-to-Image Generation — Liang et al. (2023) (arXiv:2312.10240, 2023)

What this evaluates

Evaluates how well an automatic multimodal transformer predicts fine-grained human feedback (quality scores, region-level misalignment heatmaps, and text misalignment annotations) on text-to-image generation tasks.

Datasets

• TIFA — total 100; splits: test (100)

Metrics

• correlation (primary) — range: [-1, 1]
  • Measures the statistical correlation between the model's predicted scores, heatmaps, and misalignment annotations and the corresponding human-annotated ground truth labels.

Input / output format

Input: A text prompt and its corresponding generated image.

Output: A scalar quality score, a region-level implausibility/misalignment heatmap, and a list of misaligned prompt keywords.

Scoring recipe

def compute_correlation(predictions, human_labels):
    # predictions and human_labels are arrays of floats
    # Compute Pearson correlation coefficient
    corr = np.corrcoef(predictions, human_labels)[0, 1]
    return corr

Common pitfalls

• Evaluation is strictly limited to 100 prompts due to the high cost of human annotation.
• Relies entirely on human feedback as ground truth, which may introduce annotator bias or variability.

Evidence (verbatim from paper)

For the 100 prompts for our human evaluation, they are sampled from the existing benchmark: TIFA [24]. We only did our evaluation on 100 prompts due to the high cost of the human annotation. The model predicts these annotations with high correlation to human labels.

Citation

@misc{liang2023richhumanfeedback,
  title={Rich Human Feedback for Text-to-Image Generation},
  author={Liang et al. (2023)},
  year={2023},
  note={arXiv:2312.10240}
}

• arXiv: 2312.10240