Lego Egocentric Action Eval

lego-egocentric-action-eval

LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuning — Bolin Lai et al. (2023) (arXiv:2312.03849, 2023)

What this evaluates

Evaluates a diffusion model's ability to generate egocentric action frames from a pre-action image and a text prompt. It probes the model's capacity to capture action state transitions while preserving contextual information and aligning with natural language instructions in egocentric video domains.

Datasets

• Ego4D — total 95452; splits: train (85521), test (9931)
• Epic-Kitchens-100 — total 70734; splits: train (61841), test (8893)

Metrics

• EgoVLP score (primary) — range: percent
  • Contrastive score from the EgoVLP model. The generated frame is duplicated to form a static two-frame input.
• EgoVLP+ score (primary) — range: percent
  • Contrastive score from EgoVLP. The input frame and generated frame are combined as a two-frame input to measure action state transition.
• CLIP score — range: percent
  • Standard image-to-text similarity score computed between the generated image and the text prompt using the CLIP model.
• FID — range: other
  • Fréchet Inception Distance measuring the distance between feature distributions of generated and ground truth frames. Lower is better.
• PSNR — range: other
  • Peak Signal-to-Noise Ratio measuring pixel-level fidelity between generated and ground truth frames.
• LPIPS — range: other
  • Learned Perceptual Image Patch Similarity using SqueezeNet encoder. Lower is better.
• BLIP-B / BLIP-L — range: percent
  • Image-to-text similarity: BLIP generates captions for output images, then CLIP text encoder computes text-to-text similarity with the prompt.
• User Study Win Rate — range: percent
  • Percentage of times a model's output is selected as the best by 5 AMT raters per instance, out of 300 samples per dataset.

Input / output format

Input: Egocentric image frame captured δ_i seconds before an action begins, paired with a natural language prompt (action label or enriched description).

Output: A single synthesized image frame captured δ_o seconds after the action begins.

Scoring recipe

def compute_metrics(pred_img, input_img, gt_img, prompt):
    metrics = {}
    metrics['EgoVLP'] = ego_vlp_score(duplicate(pred_img))
    metrics['EgoVLP+'] = ego_vlp_score(concat(input_img, pred_img))
    metrics['CLIP'] = clip_score(pred_img, prompt)
    metrics['FID'] = fid_score(pred_img, gt_img)
    metrics['PSNR'] = psnr_score(pred_img, gt_img)
    metrics['LPIPS'] = lpips_score(pred_img, gt_img)
    caption = blip_caption(pred_img)
    metrics['BLIP'] = clip_text_similarity(caption, prompt)
    return metrics

Common pitfalls

• Standard CLIP/BLIP metrics suffer from domain gaps when evaluating egocentric images, leading to semantic misalignment.
• Camera motion blur and high variance in body motion require aesthetic scoring and similarity filtering to select valid input/target frames.
• FID and LPIPS are lower-is-better metrics, while EgoVLP, EgoVLP+, CLIP, PSNR, and BLIP scores are higher-is-better.

Evidence (verbatim from paper)

We adopt image-to-image similarity, image-to-text similarity, and user study as metrics in our experiments. ... Importantly, given that EgoVLP is pre-trained on egocentric data and action labels, we consider EgoVLP and EgoVLP+ score as the primary automatic metrics.

Citation

@misc{lai2023lego,
  title={LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuning},
  author={Bolin Lai et al. (2023)},
  year={2023},
  note={arXiv:2312.03849}
}

• arXiv: 2312.03849