clean-text
Normalize text to handle PDF/Unicode encoding issues and security evasion. Converts Windows-1252, curly quotes, em/en dashes, ligatures, directional formatting, zero-width chars, and more to clean ASCII. Security extensions: homoglyph translation, mixed-script detection, invisible content detection, URL double-decoding, letter-spacing collapse.
Informations de source
- Dépôt
- grahama1970/agent-skills
- Dernière activité de la source
- 8 août 2026 à 13:48
- Langue détectée de SKILL.md
- anglais
- Étoiles
- 5
- Forks
- 2
Options d'installation
Le prompt qui vérifie d'abord la source est sélectionné par défaut. Vous pouvez passer à une commande directe ou télécharger une copie locale.
Vérifiez les fichiers source
Lisez SKILL.md et les fichiers associés affichés par SkillsMP avant de décider de l'installer.
Explorateur de fichiers
6 fichiersAffichage de SKILL.md
SKILL.md
Instructions source · Aperçu en lecture seule- name
- clean-text
- description
- Normalize text to handle PDF/Unicode encoding issues and security evasion. Converts Windows-1252, curly quotes, em/en dashes, ligatures, directional formatting, zero-width chars, and more to clean ASCII. Security extensions: homoglyph translation, mixed-script detection, invisible content detection, URL double-decoding, letter-spacing collapse.
- allowed-tools
- Bash, Read
- triggers
- ["clean text","normalize text","normalize unicode","fix encoding","clean pdf text","normalize pdf","text cleaning","sanitize input","homoglyph detection"]
- metadata
- {"short-description":"Clean PDF/Unicode text to ASCII + security sanitization","project":"pi-mono"}
- provides
- ["clean-text","sanitize-for-validation"]
- composes
- ["task-monitor","agentic-evals"]
- taxonomy
- ["preprocessing","text","precision","corruption"]
- disciplines
- ["extraction","data-engineering"]
# Clean Text
Comprehensive text normalization for handling PDF and Unicode encoding issues,
plus security-focused sanitization for injection detection.
## Quick Start
```bash
# Normalize text from stdin
echo "Hello\u2019world" | .pi/skills/clean-text/run.sh
# Normalize a file
.pi/skills/clean-text/run.sh document.txt
# Normalize with output file
.pi/skills/clean-text/run.sh document.txt -o clean.txt
# Treat argument as text (not filename)
.pi/skills/clean-text/run.sh -t "Hello\u201cworld\u201d"
# Show statistics
.pi/skills/clean-text/run.sh document.txt --stats
```
## What It Normalizes
| Category | Examples | Normalized To |
|----------|----------|---------------|
| **Whitespace** | Non-breaking, em/en space, hair space | Regular space |
| **Hyphens** | En dash, em dash, minus sign, figure dash | ASCII hyphen `-` |
| **Quotes** | Curly quotes, guillemets, primes | Straight `'` and `"` |
| **Windows-1252** | `\x93`, `\x94`, `\x92` | `"`, `"`, `'` |
| **Ligatures** | fi, fl, ffi, ffl | Expanded letters |
| **Bullets** | Various bullet points | Hyphen `-` |
| **Zero-width** | ZWSP, ZWNJ, ZWJ, BOM | Removed |
| **Directional** | LTR/RTL marks | Removed |
| **Control chars** | C0/C1 (except newline/tab) | Removed |
| **Line breaks** | `intro-\nduction` | `introduction` |
## Pipeline Integration
This skill is based on the same normalization used in the extractor pipeline's
s02_marker_extractor.py. The code is kept in sync with text_toolz patterns.
### Python Usage
```python
from clean_text import normalize_text
# Clean text for pattern matching
text = "1.\u00a0Introduction" # Non-breaking space
clean = normalize_text(text) # "1. Introduction"
```
## Normalization Steps
1. **Windows-1252 conversion** - Handle legacy MS Office encoding
2. **NFKC normalization** - Unicode compatibility decomposition
3. **Remove directional formatting** - LTR/RTL marks
4. **Remove control characters** - C0/C1 (preserve newlines)
5. **Normalize whitespace** - All special spaces to ASCII
6. **Normalize hyphens** - All dash variants to `-`
7. **Normalize quotes** - Curly to straight
8. **Normalize dots** - Ellipsis, leader dots
9. **Normalize bullets** - All bullet types to `-`
10. **Expand ligatures** - fi/fl/ffi/ffl
11. **Fix line-break hyphens** - Join hyphenated words
12. **Collapse whitespace** - Multiple spaces to single
## Security Sanitization API
For Tier 0 validators and injection detection, import the security functions directly:
```python
from clean_text import (
sanitize_for_validation, # NFKC + homoglyph + URL-decode + letter-spacing
has_control_chars, # null, DEL, C0 control chars
has_mixed_scripts, # Latin+Cyrillic mixing (homoglyph attack)
has_invisible_content, # >50% invisible characters
has_directional_chars, # Unicode Bidi override characters
is_whitespace_only, # empty after sanitization
HOMOGLYPH_MAP, # Cyrillic→Latin translation table
INVISIBLE_CHARS_RE, # zero-width + directional + BOM regex
)
```
Used by: `services/common/validation.py` (Embry OS Tier 0 validators)
## Based On
- text_toolz library patterns
- extractor pipeline s02 normalization
- NFKC Unicode standard
Voir sur GitHub