Vukuzenzele Za Gov Eval

vukuzenzele-za-gov-eval

Preparing the Vuk'uzenzele and ZA-gov-multilingual South African multilingual corpora — Lastrucci et al. (2023) (arXiv:2303.03750, 2023)

What this evaluates

This benchmark evaluates multilingual sentence alignment and machine translation capabilities across 11 official South African languages using government-themed corpora.

Datasets

• Vuk'uzenzele — total ?; splits: train (-1); repo https://github.com/dsfsi/vukuzenzele-nlp
• ZA-gov-multilingual — total ?; splits: train (-1); repo https://github.com/dsfsi/gov-za-multilingual

Metrics

• cosine similarity (primary) — range: [0, 1]
  • Cosine similarity between 1024-dimensional LASER sentence embeddings. Scores range from 0 to 1, with higher values indicating greater semantic similarity.

Input / output format

Input: Preprocessed source and target sentences in 11 official South African languages.

Output: CSV files named 'aligned-{src_lang_code}-{tgt_lang_code}.csv' containing source sentence, target sentence, and cosine similarity score.

Scoring recipe

# Pseudo-code
src_tokens = nltk_tokenize(preprocess(src_text))
tgt_tokens = nltk_tokenize(preprocess(tgt_text))
src_vecs = LASER.encode(src_tokens)  # 1024-dim
tgt_vecs = LASER.encode(tgt_tokens)  # 1024-dim
scores = cosine_similarity(src_vecs, tgt_vecs)
aligned = [(s, t, sc) for s, t, sc in zip(src_tokens, tgt_tokens, scores) if sc >= 0.65]

Common pitfalls

• Dataset sizes vary significantly across language pairs due to differences in scraping outcomes and source material availability.
• Preprocessing steps differ between corpora (e.g., ZA-Gov requires removing nested numbering, Vuk'uzenzele requires removing bullet points), which can affect tokenization and alignment quality.
• The fixed cosine similarity threshold of 0.65 is arbitrary and may exclude valid translations or include semantically similar but non-equivalent sentences.

Evidence (verbatim from paper)

The sentence vectors are compared and a cosine similarity algorithm was performed to produce a score from 0 to 1 on the similarity of the two vectors as described in section 4.2.1. The observations present in each dataset with a score above 0.65, or 65% similarity, are listed in the following tables 2 and 3.

Citation

@misc{lastrucci2023vukuzenzele,
  title={Preparing the Vuk'uzenzele and ZA-gov-multilingual South African multilingual corpora},
  author={Lastrucci et al. (2023)},
  year={2023},
  note={arXiv:2303.03750}
}

• arXiv: 2303.03750