Text Encoding Hygiene

Text Encoding Hygiene

Purpose

Use this skill before editing Japanese text, after seeing garbled output, or before committing changes that touched comments, translations, Markdown, JSON resources, Gradle files, or skill files. This skill is intentionally written in English so it remains readable when Japanese instructions are affected by display mojibake.

PowerShell Reading Rules

• Treat Windows PowerShell 5.1 default text I/O as unsafe for Japanese UTF-8 files.
• Before reading Japanese text in PowerShell, set:

[Console]::OutputEncoding = [System.Text.UTF8Encoding]::new($false)

• Prefer one of these explicit UTF-8 reads:

Get-Content -Encoding UTF8 path\to\file.md
[System.IO.File]::ReadAllText($path, [System.Text.Encoding]::UTF8)

• If output shows garbled Japanese, stop editing that file. Re-read it with explicit UTF-8 and confirm the text is readable before making changes.
• If a Python validation script fails while reading Japanese files under the local default encoding, set PYTHONUTF8=1 for that command.

Writing Rules

• Do not use default Set-Content or Out-File in Windows PowerShell 5.1 for repository text files.
• Prefer Codex file editing tools for source changes.
• If shell writing is unavoidable, write UTF-8 without BOM explicitly:

$utf8NoBom = [System.Text.UTF8Encoding]::new($false)
[System.IO.File]::WriteAllText($path, $text, $utf8NoBom)

• Keep edits narrow. Avoid whole-file rewrites when only a few lines need to change.

Required Checks

• Run git diff --name-status and confirm the changed files match the request.
• Inspect git diff for unintended deletion of Japanese comments or documentation.
• For code, resource, dependency, or datagen changes, run:

.\scripts\use-java.ps1
./gradlew.bat build

• For a faster encoding-only check, run:

./gradlew.bat checkTextEncodingHygiene

Mojibake Handling

• The Gradle check catches only obvious cases to avoid false positives.
• src/main/resources/assets/apprenticecodex/lang/zh_cn.json is excluded because simplified Chinese translation text is more likely to collide with simple CJK heuristics.
• If the Gradle task reports a file, re-open that file as UTF-8 and fix the source text rather than suppressing the check.
• Do not add broad exclusions. Add a narrow path exclusion only when a legitimate file is demonstrably unavoidable.

Source: hexqua/apprentice_codex — distributed by TomeVault.