# Thai Text Processing

> Thai Text Processing (ประมวลผลข้อความภาษาไทยในโค้ด)

- Skill: `boom-vitt/thai-text-processing` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx skillmds@latest add boom-vitt/thai-text-processing`
- Raw SKILL.md: https://api.skillmd.com/api/skills/boom-vitt/thai-text-processing/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: Boom-Vitt (https://skillmd.com/u/boom-vitt)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/boom-vitt/thai-text-processing

---


# Thai Text Processing (ประมวลผลข้อความภาษาไทยในโค้ด)

## Overview
Thai is written without spaces between words, uses combining tone marks, and does not sort by codepoint. Naive code that works for English silently corrupts Thai: `"ฉันกินข้าว".split(" ")` returns one token, `ORDER BY name` puts vowels in the wrong place, and `LEFT(name, 20)` may chop a syllable mid-character and render `◌`. This skill is the checklist of things to fix.

## When to use
- ตัดคำภาษาไทย (segmentation) สำหรับ search, autocomplete, n-grams
- Unicode normalization (NFC) ก่อนเก็บ / เทียบสตริง
- จัดเรียง (sort / collate) ชื่อภาษาไทยใน DB หรือ list
- ทำ romanization สำหรับ slug, passport-style spelling, academic citation
- ตั้งค่า full-text search index ใน ES / Postgres / MySQL
- ตัดข้อความ (truncate) แบบไม่ทำให้ glyph แตก
- เลือกไลบรารี: PyThaiNLP vs ICU vs nlpO3 vs Lucene Thai analyzer

## 1. The core problem: no spaces

```python
"ฉันกินข้าว".split(" ")  # ["ฉันกินข้าว"]  ← one mega-token
```

This breaks:
- full-text search (`ILIKE '%กิน%'` mostly works, `ILIKE '%ข้าว%'` mostly works, but ranking, stemming, autocomplete all need proper tokens)
- word counting / "how long is this in words"
- text wrap / hyphenation
- n-gram features for ML
- highlight-on-search snippets

You **must** run a segmenter before any token-based operation.

## 2. Word segmentation toolkit

| Tool | Language | Notes |
|---|---|---|
| **PyThaiNLP** `word_tokenize` | Python | De-facto default. Engines: `newmm` (default, dict-based), `longest`, `attacut` (CNN), `deepcut` (LSTM, slow but accurate) |
| **nlpO3** | Python via Rust | Fast newmm-style; permissive license |
| **ICU `BreakIterator`** | C / Java / Python (`pyicu`) | Built into many platforms; decent for word-break |
| **thai-segmenter** | TypeScript | Browser-friendly; dictionary-based |
| **Lucene Thai analyzer** | Java | Use for Elasticsearch / OpenSearch indexing |
| **SentencePiece / BPE** | Any | For ML training; not for human-facing tokens |

Picking rules of thumb:
- **Server Python:** PyThaiNLP `newmm` (good enough, fast, MIT)
- **Need higher recall on OOV / colloquial:** `attacut` or `deepcut` (slower)
- **Frontend JS:** `thai-segmenter` or call ICU via WebAssembly
- **Search index:** dedicated Thai analyzer in ES (`thai` tokenizer) — do not segment client-side and ship tokens to ES
- **Production at scale:** nlpO3 for raw throughput

See `examples/segmentation.py` for runnable code.

## 3. Unicode normalization (NFC)

Thai displays one "visual character" out of up to **3 codepoints**: base consonant + above-vowel + tone mark. The same word can be encoded multiple ways:

```python
import unicodedata
a = "แสง"                              # may be precomposed
b = unicodedata.normalize("NFD", a)    # explicitly decomposed
a == b                                  # False
unicodedata.normalize("NFC", a) == unicodedata.normalize("NFC", b)  # True
```

**Rules:**
- Always normalize to **NFC** before storing or comparing.
- Database column: pre-normalize in application, OR use a generated column / trigger.
- "Length in user-visible characters" = grapheme clusters, not codepoints. Use `regex` library `\X` (Python) or `Intl.Segmenter("th", {granularity: "grapheme"})` (JS).
- `len("ก่")` in Python returns 2 (codepoints) — but the user sees 1 character.

See `examples/normalize.py`.

## 4. Romanization standards

| System | Use case | Lossless? | Example: สวัสดี |
|---|---|---|---|
| **RTGS** (Royal Thai General System) | Signs, passports, SEO slugs | No (drops tone, vowel length) | `sawatdi` |
| **ISO 11940** | Academic, archives | Yes (one-to-one, ugly) | `s̄wạs̄dī` |
| **ISO 11940-2** | Phonetic, prettier | No | `sawatdi` |
| **IPA** | Phonetic transcription | Approximate | `/sā.wàt.dīː/` |

Use **RTGS** for slugs and English-readable transliteration of names. PyThaiNLP exposes `pythainlp.transliterate.romanize(text, engine="royin")` for RTGS-style output. For deterministic batch slugs, also strip non-`[a-z0-9-]` after romanization.

## 5. Sorting / collation

Thai does NOT sort by codepoint. The traditional rule: sort by **base consonant** first, then **vowel**, then **tone mark**. Codepoint sort puts leading vowels (`เ`, `แ`, `โ`, `ใ`, `ไ`) AFTER consonants, which is wrong.

| Place | How |
|---|---|
| PostgreSQL (ICU build) | `name TEXT COLLATE "th-TH-x-icu"` |
| MySQL 8 | `COLLATE utf8mb4_thai_520_w2` (Unicode 5.20, weights v2) |
| Python (no DB) | `pyicu` Collator with `Locale("th_TH")`, or PyThaiNLP's `thai_strftime`/`collate` |
| JavaScript | `Intl.Collator("th", {sensitivity: "base"})` |
| Elasticsearch | `icu_collation_keyword` field with `language: "th"` |

```sql
-- PostgreSQL example
CREATE COLLATION thai_icu (provider = icu, locale = 'th-TH');
SELECT name FROM users ORDER BY name COLLATE "th-TH-x-icu";
```

Default UTF-8 / binary sort is wrong — verify by sorting `["แอน","กิน","โต","ไกล"]` and confirming Thai dictionary order, not codepoint order.

## 6. Truncation safety

Truncating mid-character (codepoint vs grapheme) can leave dangling combining marks rendered as `◌`. Rules:

- **Truncate at grapheme cluster boundary.** Python: `regex` library `\X`. JS: `Intl.Segmenter` with `granularity: "grapheme"`.
- For UI ellipsis (e.g. "show first 30 chars + …"), first segment by word using PyThaiNLP, then count graphemes, then append `…` — don't cut mid-syllable.
- Database `VARCHAR(50)` counts **bytes** in some collations and **characters** in others. Test before assuming "50 chars" matches user perception.

See `examples/normalize.py` for grapheme-safe truncation.

## 7. Database / Elasticsearch indexing

**Elasticsearch / OpenSearch:**
```json
{
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "thai",
        "search_analyzer": "thai"
      }
    }
  }
}
```

**PostgreSQL full-text:** the bundled `tsvector` configs don't ship a Thai dictionary. Options:
- Segment in application using PyThaiNLP, store space-separated tokens in a `tsvector` column with `simple` config
- Use `pg_search` / `paradedb` with a Thai tokenizer
- Use a sidecar Elasticsearch

**Don't rely on `ILIKE '%term%'`.** Without segmentation it does substring match — works for very short queries, fails for multi-word, misses synonyms, can't rank.

## 8. LLM tokenizers and Thai text

Modern LLMs (Claude, GPT, Gemini, Llama) use BPE / SentencePiece tokenizers trained primarily on English. Thai gets a rough ride:

- Characters are frequently **split mid-cluster** — base consonant in one token, tone mark or above-vowel in another. The model still "reads" the word, but boundaries are arbitrary.
- Thai consumes **~2–4× more tokens than English** for equivalent semantic content. A 1,000-token English prompt becomes ~2,500–4,000 tokens in Thai.
- **Truncating by byte / character length can mid-split a glyph** — `โ` (leading vowel) ends up in one chunk and its consonant in the next, both render as `?` or `◌` in downstream UIs.
- **Embeddings on raw Thai degrade** vs. word-segmented input — semantic search recall drops noticeably when the tokenizer fragments words at non-linguistic boundaries.
- Char-counting tooling on top of LLM output often miscounts grapheme clusters (see §6).

**Practical advice:**
- Pre-segment with PyThaiNLP (`word_tokenize` or `Tokenizer(engine="newmm")`) before semantic search, RAG chunking, or embedding generation.
- Budget tokens at **~3× English baseline** when sizing context windows and API cost.
- **Truncate at word boundaries** (segment first, then cut), never by byte length on raw Thai.
- For chunking long Thai documents for RAG, chunk on segmented word boundaries — naive `chunk_size=512` character splits will fragment compound nouns and tank retrieval.

## 9. TIS-620 encoding (legacy)

**TIS-620** is the pre-Unicode Thai character encoding still found in:
- Old government and state-enterprise databases
- Banking core systems
- Some POS / inventory / accounting software
- Legacy CSV / Excel exports from Thai vendors
- Old PDF text layers (especially scanned-then-OCR'd)

It is a **single-byte** encoding: bytes `0x00–0x7F` are ASCII, bytes `0xA1–0xFB` map to Thai characters. Microsoft's variant is **Windows-874** / `cp874` — TIS-620 plus a few extra symbols (curly quotes, euro sign, etc.).

**Pitfalls:**
- Opening a TIS-620 file as UTF-8 yields mojibake like `àÅÂä·Â`.
- Saving the mojibake back as UTF-8 **double-encodes** — recovery requires decoding as UTF-8, re-encoding as Latin-1, then decoding as TIS-620.
- Some old systems use `0xA0` (non-breaking space) as a Thai consonant by mistake — strict TIS-620 decoders raise on it.
- File-encoding auto-detection (`chardet`, `charset-normalizer`) is **unreliable on short Thai strings** — confirm by inspecting bytes, not by guessing.

**Recipe (Python):**

```python
# Read a legacy file as TIS-620 or cp874 and write back as UTF-8
with open("legacy.csv", "rb") as f:
    raw = f.read()

# Try cp874 first (superset of TIS-620, handles MS exports too)
try:
    text = raw.decode("cp874")
except UnicodeDecodeError:
    # Fall back, replacing the stray 0xA0 etc.
    text = raw.decode("cp874", errors="replace")

with open("clean.csv", "w", encoding="utf-8") as f:
    f.write(text)

# Recover double-encoded mojibake (UTF-8 wrapper around TIS-620 bytes)
def fix_double_encoded(s: str) -> str:
    return s.encode("latin-1").decode("cp874")
```

For Pandas: `pd.read_csv("legacy.csv", encoding="cp874")`. For requests / HTTP: check `Content-Type: text/html; charset=tis-620` — old Thai gov sites still serve it.

## Common mistakes

- `.split(" ")` to count Thai words → always returns 1 — use a segmenter
- `ORDER BY name` without collation → vowels and tone marks in codepoint order, not dictionary order
- Storing un-normalized text → `WHERE name = 'แสง'` misses rows that were saved decomposed
- `LEFT(text, 20)` for "first 20 chars" → counts bytes or codepoints, not graphemes; renders `◌`
- "Username max 20 chars" enforced by codepoint count → users can submit 60-byte usernames with combining marks
- Using RTGS for accurate phonetic transcription → it drops tone and vowel length; use IPA
- Shipping client-segmented tokens to Elasticsearch → bypasses the analyzer; mismatched query-time tokenization
- Treating "ๆ" (mai-yamok, repetition mark) as a separate word → it's an in-word marker; PyThaiNLP handles it correctly, `.split()` doesn't
- Forgetting that `​` (zero-width space, U+200B) is sometimes injected by CMSs for word break hints — strip on input or normalize
- Sizing LLM API budgets by English character count → Thai costs ~3× more tokens; the prompt fits in the context window until it suddenly doesn't
- Chunking Thai documents for RAG by raw character count → fragments words mid-cluster, embeddings lose semantic meaning, retrieval recall drops
- Trusting `chardet` to identify TIS-620 vs UTF-8 on short strings → it guesses wrong on names and headlines; check the byte range manually or try `cp874` decode first
- Reading a TIS-620 file with the default UTF-8 codec and "fixing" the mojibake by hand-editing → you've now double-encoded; always re-decode from raw bytes