Windows Encoding Fixes

Windows: encodings, console, cross-platform

Experience from two coding-kit installation bug reports on Windows 10 (research.db id=141, 146). Each pitfall — with symptom, cause, fix. Apply to ANY script/file that must also work on Windows.

1. stdout encoding: cp1251 kills Russian output

Symptom: UnicodeEncodeError: '\u2713' ... codec can't encode — fails when printing ✓/✗/Cyrillic. Shows up when redirecting output (script > log 2>&1) and in a cp1251 console.

Cause: the Windows console defaults to cp1251; Python 3.12 takes the console encoding when redirecting, and Unicode does not fit into it.

Fix (required in every CLI script):

try:
    sys.stdout.reconfigure(encoding="utf-8", errors="replace")
    sys.stderr.reconfigure(encoding="utf-8", errors="replace")
except Exception:  # noqa: S110,BLE001 — reconfigure is optional
    pass

In coding-kit — the single memory/scripts/_compat.py: fix_encoding() instead of copying.

Fix for bash wrappers:

export PYTHONIOENCODING="${PYTHONIOENCODING:-utf-8}"

System-level fix (recommended for every Windows box):

[Environment]::SetEnvironmentVariable("PYTHONUTF8", "1", "User")

2. Writing files: CRLF breaks md5 checks

Symptom: Path.write_text() on Windows writes \r\n; the canonical form is \n. Any md5 check of mirrors/copies falsely fails with «files diverged».

Fix when writing: newline="\n" + byte comparison:

dst.write_text(content, encoding="utf-8", newline="\n")
# freshness check:
if dst.read_bytes() == content.encode("utf-8"): ...

Fix when checking (bash, md5 with normalization):

first=$(tr -d '\r' < "${MIRRORS[0]}" | md5sum | cut -d' ' -f1)

3. PowerShell 5.1: UTF-8 without BOM is read as cp1251

Symptom: a bootstrap .ps1 ignores --check, outputs mojibake, fails on the dash «—» in comments.

Cause: Windows PowerShell 5.1 decodes a .ps1 without BOM as cp1251; Cyrillic and dashes break parsing (down to the argument logic).

Fix: save the .ps1 as UTF-8 with BOM (EF BB BF at the start of the file).

if not data.startswith(b"\xef\xbb\xbf"):
    p.write_bytes(b"\xef\xbb\xbf" + data)

Check: head -c 3 file.ps1 | od -An -tx1 → ef bb bf.

4. npm on Windows is npm.cmd

Symptom: subprocess.run(["npm", "install", ...]) → FileNotFoundError: [WinError 2].

Cause: CreateProcess (without shell) does not run .cmd files; on Windows npm is npm.cmd.

Fix: look for npm.cmd first on Windows:

def _npm_cmd():
    if os.name == "nt":
        for name in ("npm.cmd", "npm"):
            p = shutil.which(name)
            if p:
                return p
    return "npm"

5. venv: bin vs Scripts, .exe

Symptom: the script looks for venv/bin/python — on Windows it does not exist, venv is at venv\Scripts\python.exe.

Fix (single resolver, _compat.py):

VENV_BIN = VENV / ("Scripts" if os.name == "nt" else "bin")
# the binary on Windows has .exe:
# d / "Scripts" / f"{name}.exe"  vs  d / "bin" / name

In bash: iterate candidates Scripts/python.exe and bin/python.

6. winget puts binaries not in PATH

Symptom: shutil.which("clangd") does not find it — but clangd is installed.

Cause: winget puts binaries in %LOCALAPPDATA%\Microsoft\WinGet\Links\ (this directory is not always in the process PATH).

Fix: also look there:

win_get = Path(os.environ.get("LOCALAPPDATA", HOME)) / "Microsoft" / "WinGet" / "Links"
extra = [str(win_get / "clangd.exe")] if os.name == "nt" else []

7. GitHub releases: projects use different asset formats per platform

Symptom: the script looks for win32-x64.tar.gz, but the project only ships .zip for Windows (LuaLS), or the reverse.

Fix: pick the asset format by platform:

ext = r"\.zip" if IS_NT else r"\.tar\.gz"
m = re.search(rf'https://[^"]*{key}-{variant}{ext}', json)
# unpacking: zipfile.ZipFile (NT) vs tarfile.open (posix)

8. bash in Windows PATH is a WSL stub, not Git Bash

Symptom: C:\Windows\System32\bash.exe prints «use wsl.exe --list to list distributions» and fails.

Cause: the system bash.exe is a WSL stub, not Git Bash.

Fix: in docs require Git for Windows (C:\Program Files\Git\bin\ bash.exe) or PowerShell wrappers. Do not rely on bash from PATH.

9. subprocess: child output encoding (BUG-1/4, bug report v2.4)

Symptom: UnicodeDecodeError: 'charmap' codec can't decode byte 0x98 in subprocess reader threads (part of the output is lost; for the LuaLS download stdout became None → «asset not found») or mojibake instead of «Checking mirrors» (doctor.py).

Cause: subprocess.run(..., text=True) without an explicit encoding takes the ANSI code page (locale.getencoding() — cp1251), while console children write to the OEM page (cp866, GetConsoleOutputCP). The two encodings will never match (CPython issue #105312).

Fix from both sides (shared helper + children write UTF-8):

# parent: memory/scripts/_compat.py run() — decodes utf-8 + errors=replace,
# passes PYTHONUTF8=1 to python children (they write UTF-8). NEVER fails
# on foreign encoding:
r = _compat.run(cmd, timeout=120)          # instead of subprocess.run(text=True)

# child: PowerShell 5.1 writes to pipes with the OEM page — switch at the
# start of .ps1 (about_Character_Encoding):
[Console]::OutputEncoding = [System.Text.Encoding]::UTF8
$OutputEncoding = [Console]::OutputEncoding

Rejected: encoding='oem' — only available in Python 3.13+ (we need to support 3.12); decoding via GetOEMCP() only in the parent — does not fix python-child output, which itself writes UTF-8 (after fix_encoding()), and still does not save the output when the encoding is corrupted; errors="strict" — one foreign byte drops the entire capture.

Universal cross-platform pitfalls

Our 8 pitfalls are NOT unique: they are known problem classes described in the industry (blog.shellnetsecurity.com «Cross-Platform Scripting Tips», PEP 686, pythonfriday.dev #130). Verified by web research 08.2026.

Which of our pitfalls are universal (confirmed by sources):

• stdout encoding (cp1251 vs UTF-8) — PEP 686, pythonfriday.dev #130;
• CRLF/LF when writing and in md5 checks — a classic (git core.autocrlf, .gitattributes);
• npm.cmd / .cmd files through CreateProcess — the known WinError 2 class;
• venv Scripts vs bin — a standard difference across all venv tools;
• GitHub release assets by platform — a common automation problem.

What we do NOT have, but the industry knows (add to code when relevant):

Takeaway: our pitfalls are a special case of general classes; this skill holds concrete fixes, the table above is insurance against the «next» pitfall.

Checklist «script is Windows-ready»

• fix_encoding() (or reconfigure) at the start — stdout utf-8
• child output — via _compat.run() (not subprocess.run(text=True) without encoding) — section 9; .ps1 children — [Console]::OutputEncoding
• writing files with newline="\n", byte comparison
• venv via the resolver (Scripts vs bin, .exe)
• npm → npm.cmd (or shell=True), winget links as an extra path
• .ps1 — UTF-8 with BOM
• md5 checks with tr -d '\r'
• GitHub release assets by platform (zip vs tar.gz)
• paths — only pathlib/Path.home(), no hardcoded / and \
• read/write files with explicit encoding="utf-8" (PEP 686-ready)
• temp files — tempfile.gettempdir(), not /tmp
• run python scripts/doctor.py on Windows (0 errors)

References

• memory/scripts/_compat.py — the shared cross-platform module (coding-kit).
• First Windows bug report, second, doctor diagnostics.
• Sources: blog.shellnetsecurity.com «Cross-Platform Scripting Tips and Tricks» (01.2026); PEP 686 (UTF-8 default, Python 3.15); pythonfriday.dev #130 «Different File Encodings Between Windows and Linux».