Slt Pose Eval

slt-pose-eval

Evaluation of Pose Estimation Systems for Sign Language Translation — O'Brien et al. (2026) (arXiv:2604.24609, 2026)

What this evaluates

This benchmark evaluates how different pose estimation models impact the quality of sign language translation. It probes the robustness of pose estimators to occlusion, temporal instability, and missing hand keypoints, measuring their downstream effect on translation metrics.

Datasets

• RWTH-PHOENIX-Weather 2014 — total ?; splits: test (-1)
• Signsuisse — total ?; splits: test (-1)

Metrics

• BLEU (primary) — range: [0, 100]
  • SacBLEU corpus BLEU score using signature: nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.6.0. Scores are scaled by 100.
• BLEURT — range: [0, 1]
  • Score from the BLEURT-20 pretrained model, which learns robust metrics for text generation.
• J_acc — range: other
  • Mean magnitude of the second-order temporal difference of 2D keypoints, averaged over joints and time.
• J_jerk — range: other
  • Mean magnitude of the third-order temporal difference of 2D keypoints, averaged over joints and time.
• E_v — range: other
  • Mean joint velocity magnitude, computed to contextualize jitter values with respect to overall motion.

Input / output format

Input: Video frames processed by a pose estimator to produce frame-level skeletal keypoints (.pose files). Keypoints are spatially normalized, zero-filled for missing values, flattened, and projected into a Transformer embedding space.

Output: German text translation of the sign language video.

Scoring recipe

# Translation quality
bleu = sacrebleu.corpus_bleu(preds, refs, signature="nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.6.0")
bleuurt = bleuurt20_model.evaluate(preds, refs)

# Pose stability (per sequence)
j_acc = mean(abs(second_order_diff(2d_keypoints)))
j_jerk = mean(abs(third_order_diff(2d_keypoints)))
e_v = mean(magnitude(first_order_diff(2d_keypoints)))

# Missing hand keypoints
hand_missing = sum(1 for frame in seq if mean(frame.hand_confidence == 0) >= 0.5)

Common pitfalls

• BLEU scores are reported scaled by 100 (e.g., 10.327), not as raw [0,1] probabilities.
• Jitter metrics use median and interquartile range (IQR) across sequences, not mean and standard deviation, due to right-skewed distributions.
• Missing hand keypoints are strictly defined as frames where ≥50% of hand keypoints have confidence c=0, which directly causes zero-filling in the translation input.

Evidence (verbatim from paper)

To evaluate translation quality, we report BLEU (Papineni et al., 2002) using SacreBLEU (Post, 2018)333Signature: nrefs:1|case:mixed|eff:no| tok:13a|smooth:exp|version:2.6.0 and BLEURT (Sellam et al., 2020; Pu et al., 2021) using the BLEURT-20 model.

Citation

@misc{obrien2026evaluation,
  title={Evaluation of Pose Estimation Systems for Sign Language Translation},
  author={O'Brien et al. (2026)},
  year={2026},
  note={arXiv:2604.24609}
}

• arXiv: 2604.24609