Isign Eval

isign-eval

iSign: A Benchmark for Indian Sign Language Processing — Joshi et al. (2024) (arXiv:2407.05404, 2024)

What this evaluates

Evaluates the accuracy of English text generation from Indian Sign Language (ISL) videos and pose sequences. It probes multimodal translation capabilities, specifically how well models align visual sign language signals with corresponding natural language references.

Datasets

• iSign — total 118228; splits: validation (593)

Metrics

• BLEU-4 (primary) — range: percent
  • Computes the geometric mean of modified 4-gram precisions with a brevity penalty to penalize overly short translations. Scores are scaled to 0-100.
• BLEU-1 — range: percent
  • Modified unigram precision with brevity penalty, scaled to 0-100.
• BLEU-2 — range: percent
  • Modified bigram precision with brevity penalty, scaled to 0-100.
• BLEU-3 — range: percent
  • Modified trigram precision with brevity penalty, scaled to 0-100.
• METEOR — range: percent
  • Harmonic mean of unigram precision and recall, incorporating synonymy matching and stemming, scaled to 0-100.
• WER — range: percent
  • Word Error Rate: minimum number of insertions, deletions, and substitutions required to transform the hypothesis into the reference, expressed as a percentage.
• ROUGE-L — range: percent
  • F-measure based on the longest common subsequence (LCS) between hypothesis and reference, scaled to 0-100.
• ROUGE-1 — range: percent
  • F-measure based on unigram overlap between hypothesis and reference, scaled to 0-100.
• ROUGE-2 — range: percent
  • F-measure based on bigram overlap between hypothesis and reference, scaled to 0-100.
• ROUGE-L-SUM — range: percent
  • F-measure based on LCS computed over sentence-level concatenation of hypothesis and reference, scaled to 0-100.

Input / output format

Input: Video frames or skeletal pose sequences corresponding to a signed sentence or phrase.

Output: A single English text string translating the signed input.

Scoring recipe

def compute_metrics(predictions, references):
    scores = {}
    for pred, ref in zip(predictions, references):
        scores['BLEU-4'] += nltk.translate.bleu_score.sentence_bleu([ref], pred, weights=(0.25,0.25,0.25,0.25))
        scores['METEOR'] += compute_meteor_score(pred, ref)
        scores['WER'] += edit_distance(pred, ref) / max(len(ref.split()), 1)
        scores['ROUGE-L'] += rouge_l_score(pred, ref)
    n = len(predictions)
    return {k: v * 100 / n for k, v in scores.items()}

Common pitfalls

• Only one reference translation is provided per video due to the scarcity of certified ISL signers, which may unfairly penalize models that generate valid but lexically different translations.
• Multi-signer videos are explicitly excluded from the benchmark, limiting evaluation to single-signer instructional/news content and ignoring conversational sign language dynamics.
• Metrics are reported on a small validation subset (593 pairs) rather than the full 118k dataset, so scores may not generalize to the full benchmark distribution.

Evidence (verbatim from paper)

To quantitatively estimate the reliability of the translations in the dataset, we compare the English translation text present in the dataset with the ones provided by the ISL instructors. Table [4] shows the translation scores for 593 sentences in the created dataset. Overall, the BLEU-4 score is 69.3 (indicative of high reliability), ROUGE-L Lin ([2004]) is 81.9, and WER (Word Error Rate) is 33.83.

Citation

@misc{joshi2024isign,
  title={iSign: A Benchmark for Indian Sign Language Processing},
  author={Joshi et al. (2024)},
  year={2024},
  note={arXiv:2407.05404}
}

• arXiv: 2407.05404