How do I install the Normalize skill?

Run npx skillmds@latest add majiayu000/normalize-4 in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Normalize safe to use?

This skill has not completed SkillMD's automated safety review yet. Capability flags: docs only. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Normalize?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Normalize free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Normalize?

majiayu000 (@majiayu000) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Normalize by majiayu000 — Agent Skill

Name: Normalize
Rating: 4.666666666666667 (3 reviews)
Author: majiayu000

File contents

name: normalize description: > Normalize text to handle PDF/Unicode encoding issues. Converts Windows-1252, curly quotes, em/en dashes, ligatures, directional formatting, zero-width chars, and more to clean ASCII. allowed-tools: Bash, Read triggers:

normalize text
clean text
normalize unicode
fix encoding
clean pdf text
normalize pdf metadata: short-description: Clean PDF/Unicode text to ASCII project-path: /home/graham/workspace/experiments/pi-mono provides:
text-normalization composes: [, task-monitor]

taxonomy: - preprocessing - text - precision

Text Normalize

Comprehensive text normalization for handling PDF and Unicode encoding issues.

Quick Start

# Normalize text from stdin
echo "Hello\u2019world" | .pi/skills/normalize/run.sh

# Normalize a file
.pi/skills/normalize/run.sh document.txt

# Normalize with output file
.pi/skills/normalize/run.sh document.txt -o clean.txt

# Treat argument as text (not filename)
.pi/skills/normalize/run.sh -t "Hello\u201cworld\u201d"

# Show statistics
.pi/skills/normalize/run.sh document.txt --stats

What It Normalizes

Category	Examples	Normalized To
Whitespace	Non-breaking, em/en space, hair space	Regular space
Hyphens	En dash, em dash, minus sign, figure dash	ASCII hyphen `-`
Quotes	Curly quotes, guillemets, primes	Straight `'` and `"`
Windows-1252	`\x93`, `\x94`, `\x92`	`"`, `"`, `'`
Ligatures	fi, fl, ffi, ffl	Expanded letters
Bullets	Various bullet points	Hyphen `-`
Zero-width	ZWSP, ZWNJ, ZWJ, BOM	Removed
Directional	LTR/RTL marks	Removed
Control chars	C0/C1 (except newline/tab)	Removed
Line breaks	`intro-\nduction`	`introduction`

Pipeline Integration

This skill is based on the same normalization used in the extractor pipeline's s02_marker_extractor.py. The code is kept in sync with text_toolz patterns.

Python Usage

from normalize import normalize_text

# Clean text for pattern matching
text = "1.\u00a0Introduction"  # Non-breaking space
clean = normalize_text(text)   # "1. Introduction"

Normalization Steps

Windows-1252 conversion - Handle legacy MS Office encoding
NFKC normalization - Unicode compatibility decomposition
Remove directional formatting - LTR/RTL marks
Remove control characters - C0/C1 (preserve newlines)
Normalize whitespace - All special spaces to ASCII
Normalize hyphens - All dash variants to -
Normalize quotes - Curly to straight
Normalize dots - Ellipsis, leader dots
Normalize bullets - All bullet types to -
Expand ligatures - fi/fl/ffi/ffl
Fix line-break hyphens - Join hyphenated words
Collapse whitespace - Multiple spaces to single

Based On

text_toolz library patterns
extractor pipeline s02 normalization
NFKC Unicode standard

---name: normalize-4description: Comprehensive text normalization for handling PDF and Unicode encoding issues.------name: normalizedescription: > Normalize text to handle PDF/Unicode encoding issues. Converts Windows-1252, curly quotes, em/en dashes, ligatures, directional formatting, zero-width chars, and more to clean ASCII.allowed-tools: Bash, Readtriggers: - normalize text - clean text - normalize unicode - fix encoding - clean pdf text - normalize pdfmetadata: short-description: Clean PDF/Unicode text to ASCII project-path: /home/graham/workspace/experiments/pi-monoprovides: - text-normalizationcomposes: [, task-monitor]taxonomy: - preprocessing - text - precision---# Text NormalizeComprehensive text normalization for handling PDF and Unicode encoding issues.## Quick Start```bash# Normalize text from stdinecho "Hello\u2019world" | .pi/skills/normalize/run.sh# Normalize a file.pi/skills/normalize/run.sh document.txt# Normalize with output file.pi/skills/normalize/run.sh document.txt -o clean.txt# Treat argument as text (not filename).pi/skills/normalize/run.sh -t "Hello\u201cworld\u201d"# Show statistics.pi/skills/normalize/run.sh document.txt --stats```## What It Normalizes| Category | Examples | Normalized To ||----------|----------|---------------|| **Whitespace** | Non-breaking, em/en space, hair space | Regular space || **Hyphens** | En dash, em dash, minus sign, figure dash | ASCII hyphen `-` || **Quotes** | Curly quotes, guillemets, primes | Straight `'` and `"` || **Windows-1252** | `\x93`, `\x94`, `\x92` | `"`, `"`, `'` || **Ligatures** | fi, fl, ffi, ffl | Expanded letters || **Bullets** | Various bullet points | Hyphen `-` || **Zero-width** | ZWSP, ZWNJ, ZWJ, BOM | Removed || **Directional** | LTR/RTL marks | Removed || **Control chars** | C0/C1 (except newline/tab) | Removed || **Line breaks** | `intro-\nduction` | `introduction` |## Pipeline IntegrationThis skill is based on the same normalization used in the extractor pipeline'ss02_marker_extractor.py. The code is kept in sync with text_toolz patterns.### Python Usage```pythonfrom normalize import normalize_text# Clean text for pattern matchingtext = "1.\u00a0Introduction" # Non-breaking spaceclean = normalize_text(text) # "1. Introduction"```## Normalization Steps1. **Windows-1252 conversion** - Handle legacy MS Office encoding2. **NFKC normalization** - Unicode compatibility decomposition3. **Remove directional formatting** - LTR/RTL marks4. **Remove control characters** - C0/C1 (preserve newlines)5. **Normalize whitespace** - All special spaces to ASCII6. **Normalize hyphens** - All dash variants to `-`7. **Normalize quotes** - Curly to straight8. **Normalize dots** - Ellipsis, leader dots9. **Normalize bullets** - All bullet types to `-`10. **Expand ligatures** - fi/fl/ffi/ffl11. **Fix line-break hyphens** - Join hyphenated words12. **Collapse whitespace** - Multiple spaces to single## Based On- text_toolz library patterns- extractor pipeline s02 normalization- NFKC Unicode standard

Frequently asked questions

Normalize