Ego Instructor Eval

ego-instructor-eval

Retrieval-Augmented Egocentric Video Captioning — Jilan Xu et al. (2024) (arXiv:2401.00789, 2024)

What this evaluates

This evaluation protocol assesses a retrieval-augmented egocentric video captioning framework. It probes the model's ability to perform cross-view video-text and video-video retrieval, answer multiple-choice questions based on video-text alignment, and generate accurate egocentric video captions using retrieved exocentric instructional videos as references.

Datasets

• EK100 MIR — total 9668; splits: test (9668)
• EgoMCQ — total 39751; splits: test (39751)
• SummMCQ — total 1614; splits: test (1614)
• YouCook2-Clip — total 3350; splits: test (3350)
• YouCook2-Video — total 436; splits: test (436)
• CharadesEgo — total 145; splits: test (145)
• EgoLearner-MCQ — total 1951; splits: test (1951)
• Ego4d cooking — total 7161; splits: val (7161)
• EgoLearner — total 1089; splits: test (1089)

Metrics

• mAP — range: [0, 1]
  • Mean Average Precision across all queries. Computes the average precision for each query and averages them.
• nDCG — range: [0, 1]
  • Normalized Discounted Cumulative Gain. Ranks retrieved items by relevance score, applies logarithmic discounting based on position, and normalizes by the ideal DCG.
• inter-/intra-video acc. — range: [0, 1]
  • Top-1 accuracy for multiple-choice questions where distractors are sampled from different videos (inter) or the same video (intra).
• R@1, R@5, R@10 (primary) — range: [0, 1]
  • Recall at K. The fraction of queries where the ground-truth relevant item appears in the top K retrieved results.
• BELU-4 — range: [0, 1]
  • BLEU-4 metric measuring n-gram precision up to 4-grams between generated and reference captions.
• METEOR — range: [0, 1]
  • Metric based on aligned unigrams, considering synonymy and stemming, with harmonic mean of precision and recall.
• ROUGE-L — range: [0, 1]
  • Recall-Oriented Understudy for Gisting Evaluation based on the longest common subsequence between generated and reference text.
• CIDER (primary) — range: [0, 1]
  • Consensus-based Image Description Evaluation. Computes n-gram similarity between candidate and reference captions using a consensus vector from multiple references.

Input / output format

Input: Per instance: an egocentric video clip (encoded as 4 frame features sampled at 8 fps) optionally paired with retrieved exocentric video clips and their captions. For retrieval tasks, the input is a video query against a candidate pool of videos or texts.

Output: Per instance: a ranked list of retrieved videos/texts, or a generated natural language caption describing the egocentric video action.

Scoring recipe

def score_retrieval(predictions, gold_indices, k_values=[1,5,10]):
    recalls = {}
    for k in k_values:
        top_k = predictions[:k]
        recalls[f'R@{k}'] = 1.0 if gold_indices in top_k else 0.0
    return recalls

def score_captioning(predictions, references):
    bleu4 = compute_bleu(predictions, references, n=4)
    meteor = compute_meteor(predictions, references)
    rouge_l = compute_rouge(predictions, references, 'l')
    cider = compute_cider(predictions, references)
    return {'BLEU-4': bleu4, 'METEOR': meteor, 'ROUGE-L': rouge_l, 'CIDER': cider}

Common pitfalls

• Models trained exclusively on egocentric or exocentric data fail on cross-view benchmarks due to severe view discrepancy; both views must be aligned during training.
• Using raw ASR transcripts instead of refined captions degrades retrieval and captioning performance, as refined captions better align semantic formats across views.
• Randomly selected exocentric videos provide no semantic benefit for captioning, yielding performance similar to 0-shot baselines; cross-view retrieval quality is critical.

Evidence (verbatim from paper)

We report the R@1, R@5 and R@10 for retrieval tasks and Top-1 accuracy for MCQ tasks. For video captioning, we sample 7161 cooking video clips from the Ego4d validation set. We also choose videos from EgoLearner which contains 1089 video clips with fine-grained video captions.

Citation

@misc{xu2024retrievalaugmentedegocentric,
  title={Retrieval-Augmented Egocentric Video Captioning},
  author={Jilan Xu et al. (2024)},
  year={2024},
  note={arXiv:2401.00789}
}

• arXiv: 2401.00789