Task Me Anything Eval

task-me-anything-eval

Task Me Anything — Jieyu Zhang et al. (2024) (arXiv:2406.11775, 2024)

What this evaluates

Evaluates the visual perceptual capabilities of large multimodal language models across object recognition, attribute recognition, spatial and temporal reasoning, and action recognition using programmatically generated image and video question-answering tasks.

Datasets

• Task-Me-Anything — total ?; splits: (unstated); repo https://github.com/JieyuZ2/TaskMeAnything

Metrics

• accuracy (primary) — range: [0, 1]
  • Multiple-choice accuracy: the fraction of instances where the model's predicted option exactly matches the ground truth answer.

Input / output format

Input: An image or video, a natural language question, and a list of multiple-choice options.

Output: A single selected option corresponding to the ground truth answer.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = sum(1 for pred, gold in zip(predictions, golds) if pred == gold)
    return correct / len(golds)

Common pitfalls

• Results are not static; they depend on the specific task plans sampled or the approximation method (Random, Fitting, Active) used for fine-grained queries.
• Performance varies significantly between synthetic 2D/3D rendered scenes and real-world annotated data (Visual Genome, Action Genome, GQA, AGQA).
• The benchmark supports on-budget approximation, so reported accuracy may be an estimate rather than a full evaluation unless explicitly stated.

Evidence (verbatim from paper)

We adopt the common input-output format used in existing benchmarks, i.e., all the task instances in Task-Me-Anything contain an image/video, a question, and multiple options with one ground truth answer. MLMs will be evaluated on these generated task instances and the results will be returned back to the user. Find all the object recognition tasks that both LLaVA-Next-34B and GPT4o perform below 30% accuracy?

Citation

@misc{zhang2024taskmeanything,
  title={Task Me Anything},
  author={Jieyu Zhang et al. (2024)},
  year={2024},
  note={arXiv:2406.11775}
}

• arXiv: 2406.11775