Superglasses Eval

superglasses-eval

SUPERGLASSES: Benchmarking Vision Language Models as Intelligent Agents for AI Smart Glasses — Jiang et al. (2026) (arXiv:2602.22683, 2026)

What this evaluates

This benchmark evaluates vision-language models' ability to function as intelligent agents for AI smart glasses in real-world egocentric scenarios. It probes capabilities in object detection, multi-hop reasoning, retrieval-augmented generation, and accurate answer formulation based on visual context and external knowledge.

Datasets

• SuperGlasses — total 2422; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Binary accuracy computed via an LLM-as-Judge framework (Qwen2.5-32B). A response is scored 1 if the judge determines it accurately captures all key information from the ground truth, else 0.

Input / output format

Input: An egocentric image paired with a natural language question.

Output: A natural language response/answer.

Scoring recipe

def score(predictions, gold):
    # LLM-as-Judge evaluation using Qwen2.5-32B
    prompt = f"Question: {gold['question']}\nGround Truth: {gold['answer']}\nModel Response: {predictions['answer']}"
    judge_output = qwen25_32b.generate(prompt)
    # Judge checks if response accurately captures all key information from the ground truth
    is_correct = judge_output.matches_criteria("accurately captures all key information")
    return 1.0 if is_correct else 0.0

Common pitfalls

• Naively applying heuristic RAG strategies (image-only, text-only, or multimodal) often degrades performance by introducing noise or disrupting the generation process.
• Models frequently fail to decouple multi-hop queries, leading to incorrect tool invocation or irrelevant retrieval.
• Object detection errors on egocentric images trigger incorrect visual retrieval, which severely impacts downstream answer accuracy.

Evidence (verbatim from paper)

From the results, we can make the following observations. (1) The proposed SuperGlasses poses a formidable challenge to all open-source, proprietary, and RAG-based VLMs, as even the most advanced model (i.e., Gemini 2.5 Pro) achieves only around 43% accuracy. Across difficulty levels, all models exhibit clear performance declines from Easy to Hard questions.

Citation

@misc{jiang2026superglasses,
  title={SUPERGLASSES: Benchmarking Vision Language Models as Intelligent Agents for AI Smart Glasses},
  author={Jiang et al. (2026)},
  year={2026},
  note={arXiv:2602.22683}
}

• arXiv: 2602.22683