Pca Eval

pca-eval

Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyond — Chen et al. (2023) (arXiv:2310.02071, 2023)

What this evaluates

Evaluates embodied decision-making capabilities across three dimensions: perception (visual understanding), cognition (reasoning and task breakdown), and action (executing correct steps or decisions). It tests models in autonomous driving, domestic assistance, and game-playing environments.

Datasets

• PCA-EVAL — total ?; splits: test (-1); repo https://github.com/pkunlp-icler/PCA-EVAL

Metrics

• Perception Score — range: [0, 1]
  • Human consensus score from three evaluators on a 0-1 scale assessing visual understanding and object recognition in the environment.
• Cognition Score — range: [0, 1]
  • Human consensus score from three evaluators on a 0-1 scale assessing reasoning, task decomposition, and decision logic.
• Action Score (primary) — range: [0, 1]
  • Human consensus score from three evaluators on a 0-1 scale assessing the correctness of the final decision or action execution.

Input / output format

Input: RGB image of the current environment state, plus a text prompt containing task instructions. For domestic and game domains, additional non-visual context ('items in hand', 'items in inventory') is appended to the prompt. In the HOLMES setting, pre-computed API results are also provided.

Output: Text-based reasoning and/or action decision (e.g., next step, command, or final choice). For HOLMES, a sequence of API calls and intermediate reasoning steps.

Scoring recipe

def compute_scores(predictions):
    evaluators = [HumanEvaluator1, HumanEvaluator2, HumanEvaluator3]
    scores = {'P': [], 'C': [], 'A': []}
    for pred in predictions:
        for evaluator in evaluators:
            scores['P'].append(evaluator.score_perception(pred))
            scores['C'].append(evaluator.score_cognition(pred))
            scores['A'].append(evaluator.score_action(pred))
    # Consensus score is the average across the 3 evaluators for each dimension
    return {dim: sum(scores[dim])/len(scores[dim]) for dim in ['P', 'C', 'A']}

Common pitfalls

• Open-source VLLMs often fail to output structured cross-modal reasoning, leading to unreported Perception and Cognition scores in results.
• HOLMES-style multi-step API invocation accumulates reasoning errors, causing Cognition and Action scores to drop in complex domains like Domestic and Game.
• Inter-rater reliability (kappa) is reported separately from the actual 0-1 consensus scores; do not confuse kappa values with performance scores.

Evidence (verbatim from paper)

PCA-Eval assesses embodied decision-making through three distinct lenses: perception, cognition, and action. The scores we reported in Table 1 rely on the consensus score from three human evaluators. We compute the average kappa correlation coefficient for these evaluators, resulting in 0.91 for the Perception Score and 0.88 for the Cognition Score.

Citation

@misc{chen2023towards,
  title={Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyond},
  author={Chen et al. (2023)},
  year={2023},
  note={arXiv:2310.02071}
}

• arXiv: 2310.02071