Microg 4m Eval

microg-4m-eval

Go Beyond Earth: Understanding Human Actions and Scenes in Microgravity Environments — Wen et al. (2025) (arXiv:2506.02845, 2025)

What this evaluates

Evaluates video-based human action recognition, temporal captioning, and visual question answering in microgravity environments. It probes a model's ability to generalize to orientation-invariant motion, floating objects, and lack of ground contact where terrestrial models typically fail.

Datasets

• MicroG-4M — total 4759; splits: train (3331), val (475), test (953); repo https://github.com/LEI-QI-233/HAR-in-Space

Metrics

• mAP@0.5 (primary) — range: [0, 100]
  • Mean Average Precision at Intersection-over-Union threshold 0.5, macro-averaged over all action classes. Measures detection accuracy per category.
• F1-score — range: [0, 100]
  • Macro-averaged F1 score computed per class and then averaged across all action categories.
• CIDEr — range: [0, 100]
  • Standard lexical metric for caption/QA evaluation, rescaled to a 0–100 range for consistency.
• S-BERT — range: [0, 100]
  • Cosine similarity between Sentence-BERT embeddings of predicted and reference texts, rescaled to 0–100.
• S-VQA — range: [0, 100]
  • Semantic equivalence metric for VQA answers, computed as cosine similarity between Sentence-BERT embeddings of predicted and reference answers.

Input / output format

Input: Video clips (typically sampled within a 3-second window, frame counts vary by model) for HAR and captioning; video clips paired with natural language questions for VQA.

Output: For HAR: predicted action class labels with bounding boxes/timestamps. For captioning: descriptive text captions. For VQA: natural language answers.

Scoring recipe

def compute_metrics(predictions, golds, task):
    if task == 'HAR':
        class_aps = []
        for cls in classes:
            pred_boxes = [p for p in predictions if p.label == cls]
            gold_boxes = [g for g in golds if g.label == cls]
            class_aps.append(calculate_ap(pred_boxes, gold_boxes, iou_thresh=0.5))
        return sum(class_aps) / len(class_aps) * 100
    elif task in ['captioning', 'VQA']:
        scores = {}
        scores['CIDEr'] = cider_score(predictions, golds) * 100
        scores['S-BERT'] = cosine_similarity(sbert(predictions), sbert(golds)) * 100
        if task == 'VQA':
            scores['S-VQA'] = cosine_similarity(sbert(predictions), sbert(golds)) * 100
        return scores

Common pitfalls

• Models pretrained on Earth datasets (e.g., Kinetics, AVA) degrade significantly due to gravity-dependent priors (orientation, support/contact) rather than architectural limitations.
• Lexical metrics (BLEU-4, CIDEr) drop sharply due to domain-specific vocabulary and paraphrasing, while semantic metrics (S-BERT, S-VQA) remain higher; relying solely on lexical overlap misrepresents model capability.
• Increasing input frame density within a fixed time window does not consistently improve performance; semantic salience extraction is more critical than temporal redundancy in microgravity.

Evidence (verbatim from paper)

Our evaluation metrics include mAP@0.5, F1 score, recall, and AUROC, all calculated using the macro method. Among these, mAP@0.5 is the primary metric for measuring average detection accuracy per category and thus comprehensively evaluating the model’s action recognition performance in a microgravity environment.

Citation

@misc{wen2025microg4m,
  title={Go Beyond Earth: Understanding Human Actions and Scenes in Microgravity Environments},
  author={Wen et al. (2025)},
  year={2025},
  note={arXiv:2506.02845}
}

• arXiv: 2506.02845