Grasp Eval

grasp-eval

GRASP: A novel benchmark for evaluating language GRounding And Situated Physics understanding in multimodal language models — Jassim et al. (2023) (arXiv:2311.09048, 2023)

What this evaluates

Evaluates multimodal language models' ability to understand language grounding and intuitive physics principles through video-based question answering. It probes capabilities like object detection, feature recognition, and physical plausibility reasoning using simulated Unity environments.

Datasets

• GRASP — total ?; splits: (unstated)

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly classified responses out of the total number of test instances. Calculated as (correct predictions / total predictions) * 100.

Input / output format

Input: A video clip from a Unity simulation paired with a text prompt containing a question (binary yes/no or open-ended/multi-class).

Output: For binary tasks: a response starting with 'yes' or 'no'. For open-ended/multi-class tasks: free-form text describing the observed object, feature, or physical plausibility.

Scoring recipe

def compute_accuracy(predictions, gold_labels, task_type):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        if task_type == 'binary':
            if pred.lower().startswith(('yes', 'no')):
                if pred.lower().startswith(gold.lower()):
                    correct += 1
        else:
            if normalize_tokens(pred) == normalize_tokens(gold):
                correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Binary responses are strictly invalid if they do not begin with 'yes' or 'no'; any preceding text causes the instance to be marked incorrect.
• Chance performance is not uniform across tasks: it is 50% for binary classification, but varies (25% or 50%) for multi-class classification depending on the number of answer options.
• Advanced prompting strategies like Chain-of-Thought or one-shot examples do not guarantee improvement and can significantly degrade performance for certain models.

Evidence (verbatim from paper)

For quantitative evaluation, the models’ responses are classified by a simple scheme: Responses to binary yes/no questions are only counted as valid if they begin with the word “yes” or “no”; the rest of the response is considered irrelevant. We regard responses that do not adhere to this as incorrect. Table 2: Accuracy (%) for all models on GRASP’s Level 1 using binary question (inducing binary classification) and open-ended question prompts (inducing multi-class classification).

Citation

@misc{jassim2023grasp,
  title={GRASP: A novel benchmark for evaluating language GRounding And Situated Physics understanding in multimodal language models},
  author={Jassim et al. (2023)},
  year={2023},
  note={arXiv:2311.09048}
}

• arXiv: 2311.09048