# Multi Language Document Processing

> Patterns for documents that mix Chuukese + English (and occasionally other JW.org locales) — sentence segmentation, scripture-safe chunking, and per-language sentence stores. Use when ingesting bilingual brochures/articles or building parallel corpora.

- Skill: `findinfinitelabs/multi-language-document-processing` (Agent Skill)
- Install (CLI): `npx skillmds@latest add findinfinitelabs/multi-language-document-processing`
- Raw SKILL.md: https://api.skillmd.com/api/skills/findinfinitelabs/multi-language-document-processing/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: findinfinitelabs (https://skillmd.com/u/findinfinitelabs)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/findinfinitelabs/multi-language-document-processing

---


# Multi-Language Document Processing

Most JW source material the dictionary ingests is **bilingual or interleaved** — a Chuukese paragraph followed by its English original, or scripture references that resolve to either language. This skill covers the actual patterns the repo uses; deeper Chuukese-specific guidance lives in [chuukese-language-processing](../chuukese-language-processing/SKILL.md).

## Where multilingual material lives

- [`config/data/bible/`](../../../config/data/) — `nwt_chk.epub` + `nwt_en.epub`. Same verse IDs, same chapter map. See the [bible-epub-processing](../bible-epub-processing/SKILL.md) skill.
- [`config/data/brochures/`](../../../config/data/) — JSON sentence lists per language, generated by [`scripts/extract_brochure_sentences.py`](../../../scripts/extract_brochure_sentences.py).
- `.jwpub` archives — parallel content extracted by [`scripts/extract_jwpub.py`](../../../scripts/extract_jwpub.py#L30) into per-language sentence files.

## Sentence segmentation

`IntelligentTextChunker` ([src/utils/intelligent_chunker.py](../../../src/utils/intelligent_chunker.py#L47)) carries a multi-language sentence-ending table:

```python
{
  "english":  [".", "!", "?", ";"],
  "chuukese": [".", "!", "?"],
  "general":  [".", "!", "?", ";", "。", "！", "？"],
}
```

For most extraction work, `general` is the safe default. When you know the language, prefer the language-specific list — it avoids over-splitting on English semicolons in Chuukese text.

## Scripture-safe processing

Scripture references frequently appear inside both Chuukese and English text and **must not be split**. Always wrap chunking with the protect/restore helpers from [`src/utils/scripture_parser.py`](../../../src/utils/scripture_parser.py#L77). The reference regex matches both English (`1 Cor. 13:4`) and Chuukese (`Féf. 5:42`) book aliases driven by [`config/scripture_books.json`](../../../config/scripture_books.json). See the [scripture-reference-parsing](../scripture-reference-parsing/SKILL.md) skill.

## Building parallel sentence pairs

For brochures and articles where layout pairs CHK+EN side by side:

1. Extract per-language sentence lists with [`scripts/extract_brochure_sentences.py`](../../../scripts/extract_brochure_sentences.py) or [`scripts/extract_jwpub.py`](../../../scripts/extract_jwpub.py#L30).
2. Align by **position + length ratio** — this is what the existing scripts do, with no learned alignment. It works because JW publications maintain strict sentence-by-sentence parallelism.
3. Filter pairs whose `len(en) / len(chk)` falls outside [0.5, 2.5] — those are usually misalignments.
4. Hand the pairs to [`AITrainingDataGenerator.export_training_data`](../../../src/training/ai_training_generator.py#L527) for jsonl/huggingface/ollama formats.

## Mixed-language detection

For ad-hoc text, use the same heuristic the translation engines use:

```python
import re
def detect_lang(text: str) -> str:
    has_latin = bool(re.search(r"[a-zA-Z]", text))
    has_chk_accents = bool(re.search(r"[áéíóúāēīōū]", text))
    if has_chk_accents: return "chuukese"
    if has_latin: return "english"
    return "unknown"
```

This is heuristic-only — Chuukese without diacritics looks Latin/English. For high-stakes routing, prefer langid/fasttext, but the heuristic is what the production translate endpoint uses ([app.py](../../../app.py#L1538)).

## Pitfalls

- Don't `text.lower()` Chuukese — it's safe in pure ASCII text but loses information when applied to glyphs that might combine differently across NFC/NFD.
- NFC normalize Unicode before comparison: `unicodedata.normalize("NFC", text)`. Brochure exports are inconsistent.
- The "page footer" line on JW publications usually contains language hints (`chk` / `en` slugs in URLs). Keep them — they're useful for downstream sanity checks even though they're noise for training.
- Mixing Bible-verse data with brochure data into a single training file is fine, but tag the source — quality issues bias differently per source.