# Superglasses Eval

> This benchmark evaluates vision-language models' ability to function as intelligent agents for AI smart glasses in real-world egocentric scenarios. It probes capabilities in object detection, multi-hop reasoning, retrieval-augmented generation, and accurate answer formulation based on visual context and external knowledge. Use when the user wants to benchmark on SuperGlasses, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/superglasses-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/superglasses-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/superglasses-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/superglasses-eval

---


# superglasses-eval

> SUPERGLASSES: Benchmarking Vision Language Models as Intelligent Agents for AI Smart Glasses — Jiang et al. (2026) (arXiv:2602.22683, 2026)

## What this evaluates

This benchmark evaluates vision-language models' ability to function as intelligent agents for AI smart glasses in real-world egocentric scenarios. It probes capabilities in object detection, multi-hop reasoning, retrieval-augmented generation, and accurate answer formulation based on visual context and external knowledge.

## Datasets

- **SuperGlasses** — total 2422; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Binary accuracy computed via an LLM-as-Judge framework (Qwen2.5-32B). A response is scored 1 if the judge determines it accurately captures all key information from the ground truth, else 0.

## Input / output format

**Input**: An egocentric image paired with a natural language question.

**Output**: A natural language response/answer.

## Scoring recipe

```python
def score(predictions, gold):
    # LLM-as-Judge evaluation using Qwen2.5-32B
    prompt = f"Question: {gold['question']}\nGround Truth: {gold['answer']}\nModel Response: {predictions['answer']}"
    judge_output = qwen25_32b.generate(prompt)
    # Judge checks if response accurately captures all key information from the ground truth
    is_correct = judge_output.matches_criteria("accurately captures all key information")
    return 1.0 if is_correct else 0.0
```

## Common pitfalls

- Naively applying heuristic RAG strategies (image-only, text-only, or multimodal) often degrades performance by introducing noise or disrupting the generation process.
- Models frequently fail to decouple multi-hop queries, leading to incorrect tool invocation or irrelevant retrieval.
- Object detection errors on egocentric images trigger incorrect visual retrieval, which severely impacts downstream answer accuracy.

## Evidence (verbatim from paper)

> From the results, we can make the following observations. (1) The proposed SuperGlasses poses a formidable challenge to all open-source, proprietary, and RAG-based VLMs, as even the most advanced model (i.e., Gemini 2.5 Pro) achieves only around 43% accuracy. Across difficulty levels, all models exhibit clear performance declines from Easy to Hard questions.

## Citation

```bibtex
@misc{jiang2026superglasses,
  title={SUPERGLASSES: Benchmarking Vision Language Models as Intelligent Agents for AI Smart Glasses},
  author={Jiang et al. (2026)},
  year={2026},
  note={arXiv:2602.22683}
}
```

- arXiv: 2602.22683