Progait Eval

progait-eval

ProGait: A Multi-Purpose Video Dataset and Benchmark for Transfemoral Prosthesis Users — Yin et al. (2025) (arXiv:2507.10223, 2025)

What this evaluates

Evaluates vision models on prosthesis-specific video understanding, including instance segmentation of amputees and prosthetic limbs, 2D human pose estimation with focus on lower-body keypoints, and automated gait pattern classification from pose sequences.

Datasets

• ProGait — total 412; splits: train (-1), val (-1), test (-1); repo https://github.com/pittisl/ProGait

Metrics

• mIoU (primary) — range: [0, 1]
  • Mean Intersection over Union across frames and instances. Computed as the average of intersection over union between predicted and ground truth masks.
• AP@[0.5,0.95] (primary) — range: [0, 1]
  • Average Precision computed over IoU thresholds from 0.5 to 0.95 in steps of 0.05, averaged across all 23 keypoints (or 10 lower-body keypoints).
• Top-1 Accuracy — range: [0, 1]
  • Percentage of correctly classified gait sequences out of the total test set.
• Balanced Accuracy — range: [0, 1]
  • Average of recall obtained on each class, accounting for class imbalance in the 9 gait categories.

Input / output format

Input: Video frames (frontal and sagittal views) for segmentation and pose estimation; time-series sequences of 2D pose keypoint coordinates (x,y) for gait classification.

Output: Instance segmentation masks; 2D pose keypoints (23 or 10); categorical gait labels (9 classes).

Scoring recipe

# mIoU
intersection = np.logical_and(pred_mask, gt_mask)
union = np.logical_or(pred_mask, gt_mask)
miou = np.mean(np.sum(intersection) / np.sum(union))

# AP@[0.5,0.95]
# Standard COCO AP calculation over IoU thresholds 0.5 to 0.95

# Accuracy
acc = np.mean(pred_labels == gt_labels)

Common pitfalls

• YOLO11 may output multiple masks for a single subject; evaluation requires selecting the mask with the largest intersection with ground truth or merging discrete parts.
• Grounded SAM2 performance is highly sensitive to the specific text prompt used for segmentation.
• Feeding both frontal and sagittal view pose sequences simultaneously into the gait classifier causes confusion and significantly drops accuracy compared to using sagittal view alone.

Evidence (verbatim from paper)

To ensure a fair comparison, all evaluations are done only on the test set, and all the subjects in the test set are not present in either the training set or the validation set. ... We first evaluated the $AP@[0.5,0.95]$ metric for all the 23 keypoints ... Since the YOLO11 model is originally trained for multi-object detection, tracking and instance segmentation, it can produce multiple mask outputs. To evaluate its performance in tracking a single subject, we compute the mIoU using only the predicted mask with the largest intersection with the ground truth.

Citation

@misc{yin2025progait,
  title={ProGait: A Multi-Purpose Video Dataset and Benchmark for Transfemoral Prosthesis Users},
  author={Yin et al. (2025)},
  year={2025},
  note={arXiv:2507.10223}
}

• arXiv: 2507.10223