Multi Language Document Processing

Multi-Language Document Processing

Most JW source material the dictionary ingests is bilingual or interleaved — a Chuukese paragraph followed by its English original, or scripture references that resolve to either language. This skill covers the actual patterns the repo uses; deeper Chuukese-specific guidance lives in chuukese-language-processing.

Where multilingual material lives

• config/data/bible/ — nwt_chk.epub + nwt_en.epub. Same verse IDs, same chapter map. See the bible-epub-processing skill.
• config/data/brochures/ — JSON sentence lists per language, generated by scripts/extract_brochure_sentences.py.
• .jwpub archives — parallel content extracted by scripts/extract_jwpub.py into per-language sentence files.

Sentence segmentation

IntelligentTextChunker (src/utils/intelligent_chunker.py) carries a multi-language sentence-ending table:

{
  "english":  [".", "!", "?", ";"],
  "chuukese": [".", "!", "?"],
  "general":  [".", "!", "?", ";", "。", "！", "？"],
}

For most extraction work, general is the safe default. When you know the language, prefer the language-specific list — it avoids over-splitting on English semicolons in Chuukese text.

Scripture-safe processing

Scripture references frequently appear inside both Chuukese and English text and must not be split. Always wrap chunking with the protect/restore helpers from src/utils/scripture_parser.py. The reference regex matches both English (1 Cor. 13:4) and Chuukese (Féf. 5:42) book aliases driven by config/scripture_books.json. See the scripture-reference-parsing skill.

Building parallel sentence pairs

For brochures and articles where layout pairs CHK+EN side by side:

1. Extract per-language sentence lists with scripts/extract_brochure_sentences.py or scripts/extract_jwpub.py.
2. Align by position + length ratio — this is what the existing scripts do, with no learned alignment. It works because JW publications maintain strict sentence-by-sentence parallelism.
3. Filter pairs whose len(en) / len(chk) falls outside [0.5, 2.5] — those are usually misalignments.
4. Hand the pairs to AITrainingDataGenerator.export_training_data for jsonl/huggingface/ollama formats.

Mixed-language detection

For ad-hoc text, use the same heuristic the translation engines use:

import re
def detect_lang(text: str) -> str:
    has_latin = bool(re.search(r"[a-zA-Z]", text))
    has_chk_accents = bool(re.search(r"[áéíóúāēīōū]", text))
    if has_chk_accents: return "chuukese"
    if has_latin: return "english"
    return "unknown"

This is heuristic-only — Chuukese without diacritics looks Latin/English. For high-stakes routing, prefer langid/fasttext, but the heuristic is what the production translate endpoint uses (app.py).

Pitfalls

• Don't text.lower() Chuukese — it's safe in pure ASCII text but loses information when applied to glyphs that might combine differently across NFC/NFD.
• NFC normalize Unicode before comparison: unicodedata.normalize("NFC", text). Brochure exports are inconsistent.
• The "page footer" line on JW publications usually contains language hints (chk / en slugs in URLs). Keep them — they're useful for downstream sanity checks even though they're noise for training.
• Mixing Bible-verse data with brochure data into a single training file is fine, but tag the source — quality issues bias differently per source.