Bengali Asr Diarization Eval

bengali-asr-diarization-eval

Make It Hard to Hear, Easy to Learn: Long-Form Bengali ASR and Speaker Diarization via Extreme Augmentation and Perfect Alignment — Hasan et al. (2026) (arXiv:2602.23070, 2026)

What this evaluates

Evaluates automatic speech recognition (ASR) accuracy and speaker diarization performance on long-form Bengali speech. It measures phonetic robustness and computational efficiency using public and private test splits under strict hardware constraints.

Datasets

• Lipi-Ghor-882 — total 882; splits: test (22)

Metrics

• WER (primary) — range: percent
  • Word Error Rate: 100 * (Substitutions + Deletions + Insertions) / Total Reference Words.
• DER — range: percent
  • Diarization Error Rate: 100 * (False Alarms + Missed Detections + Speaker Confusions) / Total Speech Duration.
• RTF — range: ratio
  • Real-Time Factor: Inference Processing Time / Audio Duration.

Input / output format

Input: Long-form Bengali audio recordings from public and private test sets.

Output: ASR: Transcribed text. Diarization: Speaker turn labels and timestamps.

Scoring recipe

def compute_wer(hypothesis, reference):
    return 100 * (substitutions + deletions + insertions) / len(reference)

def compute_der(pred_spk_assignments, gt_spk_assignments):
    return 100 * (false_alarms + missed_detections + confusions) / total_duration

def compute_rtf(process_time, audio_duration):
    return process_time / audio_duration

Common pitfalls

• Public and private test splits are evaluated separately; results must be reported for both.
• RTF is measured under strict hardware constraints (2x T4 GPUs for inference), making cross-hardware comparisons invalid.
• Diarization accuracy relies on heuristic post-processing of baseline outputs rather than end-to-end model retraining.

Evidence (verbatim from paper)

Trainings were done using L40S GPU, 48GB VRAM. All inference processes were evaluated under strict hardware constraints (2x T4 GPUs). The separation of the ASR and Diarization pipelines allowed for maximum computational efficiency without relying on complex, end-to-end joint modeling.

TABLE II: ASR Performance Benchmarks on 22-Hours Test Set

TABLE III: Speaker Diarization Benchmarks

| Model | Retrained | Post-Proc | Pub DER | Priv DER | RTF |

Citation

@misc{hasan2026bengali,
  title={Make It Hard to Hear, Easy to Learn: Long-Form Bengali ASR and Speaker Diarization via Extreme Augmentation and Perfect Alignment},
  author={Hasan et al. (2026)},
  year={2026},
  note={arXiv:2602.23070}
}

• arXiv: 2602.23070