Quranic Audio Annotation Eval

quranic-audio-annotation-eval

Quranic Audio Dataset: Crowdsourced and Labeled Recitation from Non-Arabic Speakers — Salameh et al. (2024) (arXiv:2405.02675, 2024)

What this evaluates

Evaluates the reliability and accuracy of crowdsourced annotations for Quranic recitation audio. It measures annotator performance against expert labels, assesses inter-rater consistency, and validates an automated label-selection algorithm.

Datasets

• Quranic Audio Dataset — total 7000; splits: labeled (1166)

Metrics

• Matthews Correlation Coefficient (MCC) (primary) — range: [-1, 1]
  • Measures multiclass classification quality by considering true/false positives and negatives. Formula: (TP×TN − FP×FN) / √((TP+FP)(TP+FN)(TN+FP)(TN+FN)).
• Accuracy — range: [0, 1]
  • Ratio of correctly predicted labels to total predictions.
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall, computed macro-averaged across classes.
• Krippendorff's alpha — range: [-1, 1]
  • Statistic for measuring inter-rater reliability for any number of raters and nominal data. Values range from -1 to 1, with higher indicating better agreement.

Input / output format

Input: Audio recording of a Quranic verse or Dua recitation.

Output: One of six categorical labels: 'Correct', 'Has mistake', 'Incomplete Verse', 'Different Verse', 'Multiple Verses', or 'Empty / Not related'.

Scoring recipe

def evaluate(preds, golds, annotator_matrix):
    acc = accuracy_score(golds, preds)
    f1 = f1_score(golds, preds, average='macro')
    mcc = matthews_corrcoef(golds, preds)
    alpha = krippendorff.alpha(reliability_data=annotator_matrix, level_of_measurement='nominal')
    return {'accuracy': acc, 'f1': f1, 'mcc': mcc, 'krippendorff_alpha': alpha}

Common pitfalls

• Annotators frequently confuse semantically similar categories, particularly 'Multiple Verses' vs 'Different Verse' and 'Correct' vs 'Multiple Verses'.
• Poor audio quality or overlapping recitations cause ambiguous labels, highlighting the need for a dedicated 'Not Clear' category.
• Annotator qualification uses a strict MCC ≥ 0.6 threshold on an 8-question entrance exam, which may filter out borderline but capable contributors.

Evidence (verbatim from paper)

To calculate how well the participants performs while solving the task we used the Matthews correlation coefficient (MCC) metric since it is a more reliable statistical rate that produces a high score only if the prediction obtained good results in all of the four confusion matrix categories (true positives, false negatives, true negatives, and false positives). To pass the entrance exam, they should score MCC ≥ 0.6. This number was picked as a threshold after experiments for the number of mistakes that annotators can make in the different classes. A participant will not pass if they make more than one mistake in labeling Correct or Has Mistakes tasks or make more than two mistakes in labeling the rest of the labels. Based on the data presented in the given table III, we can conclude that allowing users to make three attempts was a suitable number. By computing the average Matthews Correlation Coefficient (MCC), Accuracy, and F1 score of individual annotators, the overall quality of the participants was evaluated. The analysis yielded an estimated overall quality with an AVG_MCC of 0.68, AVG_Accuracy of 0.77, and AVG_F1score of 0.74.

Citation

@misc{salameh2024quranicdataset,
  title={Quranic Audio Dataset: Crowdsourced and Labeled Recitation from Non-Arabic Speakers},
  author={Salameh et al. (2024)},
  year={2024},
  note={arXiv:2405.02675}
}

• arXiv: 2405.02675