Encoding Guardian

Encoding Guardian v2.0

Prevents AI coding agents (Codex, Claude Code, Roo, Cline, etc.) from silently corrupting legacy source files by stripping BOM or re-encoding GBK content to plain UTF-8.

Requirements: Python 3.8+; optional: pip install chardet for improved Latin-1 / low-confidence detection.

Core Rule

apply_patch and most file-write tools always output UTF-8 no-BOM. For any file whose original encoding is NOT UTF-8 no-BOM, you MUST snapshot before and restore after every agent edit.

Supported encodings:

Quick Reference

python scripts/enc_guard.py batch-snapshot <project_dir>
python scripts/enc_guard.py batch <project_dir>
python scripts/enc_guard.py status <project_dir>
python scripts/enc_guard.py clean <project_dir>

python scripts/enc_guard.py detect   <file>
python scripts/enc_guard.py snapshot <file>
# ... edit ...
python scripts/enc_guard.py restore  <file>
python scripts/enc_guard.py verify   <file>

Workflow: Editing a Single File

Step 1 — Detect before touching

python scripts/enc_guard.py detect <file>

Act on the result:

• UTF-8 → safe to edit normally, skip snapshot/restore
• Anything else (GBK, UTF-8-BOM, BIG5, SHIFT_JIS, UTF-16-*…) → snapshot is mandatory

Step 2 — Snapshot encoding metadata

python scripts/enc_guard.py snapshot <file>
python scripts/enc_guard.py snapshot <file> --force

Creates <file>.enc_meta alongside the source and stores encoding label, BOM flag, and original SHA-256.

Step 3 — Apply the edit

Proceed with apply_patch or other write tools.

Step 4 — Restore original encoding

python scripts/enc_guard.py restore <file>

This re-encodes the agent-written UTF-8 content back to the original encoding using an atomic write.

Step 5 — Verify

python scripts/enc_guard.py verify <file>

Check encoding match and whether file content changed relative to the snapshot.

Workflow: Batch Session

python scripts/enc_guard.py batch-snapshot <project_dir>
python scripts/enc_guard.py batch-snapshot <project_dir> --dry-run
python scripts/enc_guard.py batch-snapshot <project_dir> --ext .cpp .h .hlsl .lua .usf .ush
python scripts/enc_guard.py batch-snapshot <project_dir> --force

# ... do edits ...

python scripts/enc_guard.py batch <project_dir>
python scripts/enc_guard.py batch <project_dir> --dry-run
python scripts/enc_guard.py batch <project_dir> --no-verify

Workflow: Emergency Fix

If a file is already corrupted:

python scripts/enc_guard.py fix <file> --enc GBK
python scripts/enc_guard.py fix <file> --enc GBK --from-enc utf-8-sig
python scripts/enc_guard.py fix <file> --enc UTF-8-BOM

svn cat -r PREV <file> > original.bak
python scripts/enc_guard.py detect original.bak

Use latin-1 passthrough during recovery when the current bytes must be preserved exactly before re-encoding.

Workflow: Audit and Inspection

python scripts/enc_guard.py scan <project_dir>
python scripts/enc_guard.py scan <project_dir> --filter GBK
python scripts/enc_guard.py scan <project_dir> --filter UTF-8-BOM
python scripts/enc_guard.py scan <project_dir> --min-confidence 0.85
python scripts/enc_guard.py scan <project_dir> --ext .cpp .h .hlsl .usf .ush .lua
python scripts/enc_guard.py scan <project_dir> --json > encoding_audit.json

Workflow: Session Cleanup

python scripts/enc_guard.py clean <project_dir> --dry-run
python scripts/enc_guard.py clean <project_dir>

For SVN projects, ignore *.enc_meta to avoid working-copy noise.

Integration with apply_patch

When editing legacy files:

[ ] detect encoding
[ ] if encoding != UTF-8 no-BOM: snapshot
[ ] edit with apply_patch or write tool
[ ] if original encoding != UTF-8 no-BOM: restore
[ ] verify encoding and SHA state

Never skip restore for GBK or UTF-8-BOM files.

Edge Cases

HLSL / GLSL / USF / USH shaders

Often UTF-8-BOM in VS-driven pipelines. Snapshot these before edits.

.ini / .cfg

Often GBK in legacy Chinese projects. Snapshot is mandatory.

Binary files

enc_guard.py skips binary or undetectable files automatically.

Sparse CJK content

Detection confidence can be lower. Install chardet when confidence is weak.

Paths with spaces

Always quote paths.

Interrupted writes

The script uses temp-file + rename for atomic writes.

Script Reference

enc_guard.py detect         <file>
enc_guard.py snapshot       <file> [--force]
enc_guard.py restore        <file> [--no-verify]
enc_guard.py verify         <file>
enc_guard.py fix            <file> --enc GBK [--from-enc latin-1]
enc_guard.py scan           <dir>  [--ext ...] [--filter ENC] [--json] [--min-confidence N]
enc_guard.py batch-snapshot <dir>  [--ext ...] [--dry-run] [--force]
enc_guard.py batch          <dir>  [--dry-run] [--no-verify]
enc_guard.py status         <dir>
enc_guard.py clean          <dir>  [--dry-run]

Exit Codes

Exit code 2 is non-fatal so batch operations can continue while flagging files for review.