Kencorpus QA Eval

kencorpus-qa-eval

Kencorpus: A Kenyan Language Corpus of Swahili, Dholuo and Luhya for Natural Language Processing Tasks — Wanjawa et al. (2022) (arXiv:2208.12081, 2022)

What this evaluates

Evaluates machine reading comprehension and question answering on low-resource Kiswahili. It tests a model's ability to read short stories and accurately extract or generate answers to posed questions. The protocol uses a held-out test split to measure token-level overlap and exact string matching against ground truth answers.

Datasets

• KenSwQuAD — total 7537; splits: train (400), test (100)

Metrics

• F1 score (primary) — range: [0, 1]
  • Token-level F1 score computed between the predicted answer span and the ground truth answer.
• Exact Match (EM) — range: [0, 1]
  • Binary indicator that is 1 if the predicted answer exactly matches the ground truth string, 0 otherwise.
• Word Error Rate (WER) — range: [0, 1]
  • Ratio of (insertions + deletions + substitutions) to the total number of reference words in the speech transcript.

Input / output format

Input: A Kiswahili story text and a question derived from it.

Output: A predicted answer string extracted from or generated for the story.

Scoring recipe

def compute_metrics(predictions, golds):
    f1_scores = []
    em_scores = []
    for pred, gold in zip(predictions, golds):
        pred_tokens = pred.lower().split()
        gold_tokens = gold.lower().split()
        common = Counter(pred_tokens) & Counter(gold_tokens)
        num_same = sum(common.values())
        if num_same == 0:
            f1_scores.append(0)
            em_scores.append(0)
            continue
        precision = num_same / len(pred_tokens)
        recall = num_same / len(gold_tokens)
        f1 = 2 * precision * recall / (precision + recall)
        f1_scores.append(f1)
        em_scores.append(1.0 if pred == gold else 0.0)
    return sum(f1_scores)/len(f1_scores), sum(em_scores)/len(em_scores)

Common pitfalls

• The evaluation uses a very small test set (100 QA pairs from 100 stories), which may lead to high variance in reported metrics.
• Two different evaluation methods are reported (deep learning vs. semantic network) using different metrics (F1 vs. EM), making direct comparison difficult.
• The STT WER is reported on a separate small speech subset (27.5 hours), not the full corpus.

Evidence (verbatim from paper)

The first proof of concept that tested a QA system on the newly developed QA dataset was based on a deep learning system, specifically XLM-RoBERTa... using a dataset of 100 stories with 500 QA pairs from the Kencorpus QA dataset. The model used 80% of the data for training, while 20% of the data was used for testing... The summary of performance of the two proof of concept systems developed for the project is shown in Table 4.6, with EM being exact match, while WER being word error rate.

Citation

@misc{wanjawa2022kencorpus,
  title={Kencorpus: A Kenyan Language Corpus of Swahili, Dholuo and Luhya for Natural Language Processing Tasks},
  author={Wanjawa et al. (2022)},
  year={2022},
  note={arXiv:2208.12081}
}

• arXiv: 2208.12081