Humanref Eval

humanref-eval

Referring to Any Person — Qing Jiang et al. (2025) (arXiv:2503.08507, 2025)

What this evaluates

Evaluates a model's ability to detect all instances of a person matching a natural language description in an image, including handling multiple instances and correctly rejecting cases where the described person is absent.

Datasets

• HumanRef — total ?; splits: test (-1); repo https://github.com/IDEA-Research/RexSeek

Metrics

• Precision — range: [0, 1]
  • Fraction of predicted bounding boxes that correctly match a ground truth instance (IoU > threshold).
• Recall — range: [0, 1]
  • Fraction of ground truth instances that are successfully matched by at least one predicted bounding box.
• DensityF1 Score (primary) — range: [0, 1]
  • Average F1 score across instances weighted by a density penalty factor: DensityF1 = (1/N) * Σ [2*(P_i*R_i)/(P_i+R_i) * D_i], where D_i = min(1.0, GT_Count_i / Predicted_Count_i). Penalizes over-detection.
• Rejection Score — range: percent
  • Percentage of referring expressions for which the model predicts zero bounding boxes, measuring hallucination/rejection capability.

Input / output format

Input: An image and a natural language referring expression describing one or more persons.

Output: One or more bounding boxes (or a single point for point-output models) indicating the location(s) of the referred person(s).

Scoring recipe

def compute_metrics(predictions, gold, iou_thresholds=range(0.5, 0.96, 0.05)):
    ious = compute_iou(predictions, gold)
    matched = ious.max(axis=0) > iou_thresholds
    precision = matched.sum() / max(len(predictions), 1)
    recall = matched.sum() / max(len(gold), 1)
    f1 = 2 * precision * recall / (precision + recall + 1e-8)
    density_penalty = min(1.0, len(gold) / max(len(predictions), 1))
    density_f1 = f1 * density_penalty
    return precision, recall, density_f1

Common pitfalls

• Models trained on single-instance datasets (e.g., RefCOCO) severely drop in recall when referring to multiple persons.
• Point-in-mask evaluation is less strict than IoU-based evaluation, making direct comparisons unfair.
• Models often hallucinate bounding boxes even when the described person is absent, leading to low rejection scores.

Evidence (verbatim from paper)

We evaluate the referring task using Precision, Recall, and DensityF1 Score. Given a referring expression, the model predicts one or more bounding boxes, and a prediction is considered correct if its IoU with any ground truth box exceeds a predefined threshold. Following the evaluation protocol in COCO[[37]], we report the average performance across IoU thresholds from 0.5 to 0.95 in increments of 0.05. For the rejection subset, we calculate the number of referring expressions that the model does not predict any boxes and divide it by the number of total expressions.

Citation

@misc{jiang2025referringtoanyperson,
  title={Referring to Any Person},
  author={Qing Jiang et al. (2025)},
  year={2025},
  note={arXiv:2503.08507}
}

• arXiv: 2503.08507