His Bench Eval

his-bench-eval

HIS-GPT: Towards 3D Human-In-Scene Multimodal Understanding — Jiahe Zhao et al. (2025) (arXiv:2503.12955, 2025)

What this evaluates

Evaluates a model's ability to perform 3D human-in-scene multimodal understanding through open-ended question answering. It probes capabilities in activity recognition, spatial relationship reasoning, and human-object interaction analysis within dynamic 3D environments.

Datasets

• HIS-Bench — total ?; splits: test (-1); repo https://github.com/ZJHTerry18/HumanInScene

Metrics

• HIS-Bench score (primary) — range: percent
  • Each open-ended answer is evaluated by GPT-4 on a scale of 0 to 2. Scores are summed across 50 questions per task, yielding a maximum of 100 per task. Sub-metrics track Activity (Act.), Spatial Relationship (Spa.), and Human-Object Interaction (HoI.).

Input / output format

Input: 3D scene mesh and sequential human motion data (provided as point clouds or rendered video frames), paired with open-ended natural language questions about human-scene interactions.

Output: Open-ended natural language text answers.

Scoring recipe

def score_his_bench(predictions, gold_questions):
    task_scores = []
    current_task_scores = []
    for pred, question in zip(predictions, gold_questions):
        score = call_gpt4_evaluator(question, pred)  # Returns 0, 1, or 2
        current_task_scores.append(score)
        if len(current_task_scores) == 50:
            task_scores.append(sum(current_task_scores))
            current_task_scores = []
    return task_scores

Common pitfalls

• Open-ended answers require LLM-as-a-judge evaluation, which can introduce bias or inconsistency compared to exact-match metrics.
• Baselines often render 3D data to 2D videos, losing critical depth and occlusion information that the proposed model handles natively.
• GPT-4 scoring thresholds (0-2) depend heavily on the specific prompt template used, making cross-paper comparisons difficult without identical prompts.

Evidence (verbatim from paper)

Considering that HIS-Bench consists of open-ended questions, we use GPT-4 as an automatic evaluator to assess answer correctness. Following[[15]], we prompt GPT-4 to assign a score between 0 and 2 for each answer. Since each task in HIS-Bench consists of 50 questions, the full score for each task is 100.

Citation

@misc{zhao2025hisgpt,
  title={HIS-GPT: Towards 3D Human-In-Scene Multimodal Understanding},
  author={Jiahe Zhao et al. (2025)},
  year={2025},
  note={arXiv:2503.12955}
}

• arXiv: 2503.12955