# Lego Egocentric Action Eval

> Evaluates a diffusion model's ability to generate egocentric action frames from a pre-action image and a text prompt. It probes the model's capacity to capture action state transitions while preserving contextual information and aligning with natural language instructions in egocentric video domains. Use when the user wants to benchmark on Ego4D, Epic-Kitchens-100, or asks about evaluating this task. Reports EgoVLP score, EgoVLP+ score.

- Skill: `qhjqhj00/lego-egocentric-action-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/lego-egocentric-action-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/lego-egocentric-action-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/lego-egocentric-action-eval

---


# lego-egocentric-action-eval

> LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuning — Bolin Lai et al. (2023) (arXiv:2312.03849, 2023)

## What this evaluates

Evaluates a diffusion model's ability to generate egocentric action frames from a pre-action image and a text prompt. It probes the model's capacity to capture action state transitions while preserving contextual information and aligning with natural language instructions in egocentric video domains.

## Datasets

- **Ego4D** — total 95452; splits: train (85521), test (9931)
- **Epic-Kitchens-100** — total 70734; splits: train (61841), test (8893)

## Metrics

- `EgoVLP score` **(primary)** — range: percent
  - Contrastive score from the EgoVLP model. The generated frame is duplicated to form a static two-frame input.
- `EgoVLP+ score` **(primary)** — range: percent
  - Contrastive score from EgoVLP. The input frame and generated frame are combined as a two-frame input to measure action state transition.
- `CLIP score` — range: percent
  - Standard image-to-text similarity score computed between the generated image and the text prompt using the CLIP model.
- `FID` — range: other
  - Fréchet Inception Distance measuring the distance between feature distributions of generated and ground truth frames. Lower is better.
- `PSNR` — range: other
  - Peak Signal-to-Noise Ratio measuring pixel-level fidelity between generated and ground truth frames.
- `LPIPS` — range: other
  - Learned Perceptual Image Patch Similarity using SqueezeNet encoder. Lower is better.
- `BLIP-B / BLIP-L` — range: percent
  - Image-to-text similarity: BLIP generates captions for output images, then CLIP text encoder computes text-to-text similarity with the prompt.
- `User Study Win Rate` — range: percent
  - Percentage of times a model's output is selected as the best by 5 AMT raters per instance, out of 300 samples per dataset.

## Input / output format

**Input**: Egocentric image frame captured δ_i seconds before an action begins, paired with a natural language prompt (action label or enriched description).

**Output**: A single synthesized image frame captured δ_o seconds after the action begins.

## Scoring recipe

```python
def compute_metrics(pred_img, input_img, gt_img, prompt):
    metrics = {}
    metrics['EgoVLP'] = ego_vlp_score(duplicate(pred_img))
    metrics['EgoVLP+'] = ego_vlp_score(concat(input_img, pred_img))
    metrics['CLIP'] = clip_score(pred_img, prompt)
    metrics['FID'] = fid_score(pred_img, gt_img)
    metrics['PSNR'] = psnr_score(pred_img, gt_img)
    metrics['LPIPS'] = lpips_score(pred_img, gt_img)
    caption = blip_caption(pred_img)
    metrics['BLIP'] = clip_text_similarity(caption, prompt)
    return metrics
```

## Common pitfalls

- Standard CLIP/BLIP metrics suffer from domain gaps when evaluating egocentric images, leading to semantic misalignment.
- Camera motion blur and high variance in body motion require aesthetic scoring and similarity filtering to select valid input/target frames.
- FID and LPIPS are lower-is-better metrics, while EgoVLP, EgoVLP+, CLIP, PSNR, and BLIP scores are higher-is-better.

## Evidence (verbatim from paper)

> We adopt image-to-image similarity, image-to-text similarity, and user study as metrics in our experiments. ... Importantly, given that EgoVLP is pre-trained on egocentric data and action labels, we consider EgoVLP and EgoVLP+ score as the primary automatic metrics.

## Citation

```bibtex
@misc{lai2023lego,
  title={LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuning},
  author={Bolin Lai et al. (2023)},
  year={2023},
  note={arXiv:2312.03849}
}
```

- arXiv: 2312.03849