clean-text
Normalize text to handle PDF/Unicode encoding issues and security evasion. Converts Windows-1252, curly quotes, em/en dashes, ligatures, directional formatting, zero-width chars, and more to clean ASCII. Security extensions: homoglyph translation, mixed-script detection, invisible content detection, URL double-decoding, letter-spacing collapse.
معلومات المصدر
- المستودع
- grahama1970/agent-stack-public
- آخر نشاط في المصدر
- ٢٤ سبتمبر ٢٠٢٦ في ١٥:٥١
- لغة SKILL.md المكتشفة
- الإنجليزية
- النجوم
- ٠
- التفرعات
- ٠
خيارات التثبيت
يُحدَّد Prompt الذي يراجع المصدر أولًا بشكل افتراضي. يمكنك التبديل إلى أمر مباشر أو تنزيل نسخة محلية.
مراجعة ملفات المصدر
اقرأ SKILL.md وأي ملفات مرافقة يعرضها SkillsMP قبل أن تقرر التثبيت.
مستكشف الملفات
5 ملفاتعرض SKILL.md
SKILL.md
تعليمات المصدر · معاينة للقراءة فقط- name
- clean-text
- description
- Normalize text to handle PDF/Unicode encoding issues and security evasion. Converts Windows-1252, curly quotes, em/en dashes, ligatures, directional formatting, zero-width chars, and more to clean ASCII. Security extensions: homoglyph translation, mixed-script detection, invisible content detection, URL double-decoding, letter-spacing collapse.
- allowed-tools
- Bash, Read
- triggers
- ["clean text","normalize text","normalize unicode","fix encoding","clean pdf text","normalize pdf","text cleaning","sanitize input","homoglyph detection"]
- metadata
- {"short-description":"Clean PDF/Unicode text to ASCII + security sanitization","project":"pi-mono"}
- provides
- ["clean-text","sanitize-for-validation"]
- composes
- ["task-monitor","agentic-evals"]
- taxonomy
- ["preprocessing","text","precision","corruption"]
- disciplines
- ["extraction","data-engineering"]
# Clean Text
Comprehensive text normalization for handling PDF and Unicode encoding issues,
plus security-focused sanitization for injection detection.
## Quick Start
```bash
# Normalize text from stdin
echo "Hello\u2019world" | .pi/skills/clean-text/run.sh
# Normalize a file
.pi/skills/clean-text/run.sh document.txt
# Normalize with output file
.pi/skills/clean-text/run.sh document.txt -o clean.txt
# Treat argument as text (not filename)
.pi/skills/clean-text/run.sh -t "Hello\u201cworld\u201d"
# Show statistics
.pi/skills/clean-text/run.sh document.txt --stats
```
## What It Normalizes
| Category | Examples | Normalized To |
|----------|----------|---------------|
| **Whitespace** | Non-breaking, em/en space, hair space | Regular space |
| **Hyphens** | En dash, em dash, minus sign, figure dash | ASCII hyphen `-` |
| **Quotes** | Curly quotes, guillemets, primes | Straight `'` and `"` |
| **Windows-1252** | `\x93`, `\x94`, `\x92` | `"`, `"`, `'` |
| **Ligatures** | fi, fl, ffi, ffl | Expanded letters |
| **Bullets** | Various bullet points | Hyphen `-` |
| **Zero-width** | ZWSP, ZWNJ, ZWJ, BOM | Removed |
| **Directional** | LTR/RTL marks | Removed |
| **Control chars** | C0/C1 (except newline/tab) | Removed |
| **Line breaks** | `intro-\nduction` | `introduction` |
## Pipeline Integration
This skill is based on the same normalization used in the extractor pipeline's
s02_marker_extractor.py. The code is kept in sync with text_toolz patterns.
### Python Usage
```python
from clean_text import normalize_text
# Clean text for pattern matching
text = "1.\u00a0Introduction" # Non-breaking space
clean = normalize_text(text) # "1. Introduction"
```
## Normalization Steps
1. **Windows-1252 conversion** - Handle legacy MS Office encoding
2. **NFKC normalization** - Unicode compatibility decomposition
3. **Remove directional formatting** - LTR/RTL marks
4. **Remove control characters** - C0/C1 (preserve newlines)
5. **Normalize whitespace** - All special spaces to ASCII
6. **Normalize hyphens** - All dash variants to `-`
7. **Normalize quotes** - Curly to straight
8. **Normalize dots** - Ellipsis, leader dots
9. **Normalize bullets** - All bullet types to `-`
10. **Expand ligatures** - fi/fl/ffi/ffl
11. **Fix line-break hyphens** - Join hyphenated words
12. **Collapse whitespace** - Multiple spaces to single
## Security Sanitization API
For Tier 0 validators and injection detection, import the security functions directly:
```python
from clean_text import (
sanitize_for_validation, # NFKC + homoglyph + URL-decode + letter-spacing
has_control_chars, # null, DEL, C0 control chars
has_mixed_scripts, # Latin+Cyrillic mixing (homoglyph attack)
has_invisible_content, # >50% invisible characters
has_directional_chars, # Unicode Bidi override characters
is_whitespace_only, # empty after sanitization
HOMOGLYPH_MAP, # Cyrillic→Latin translation table
INVISIBLE_CHARS_RE, # zero-width + directional + BOM regex
)
```
Used by: `services/common/validation.py` (Embry OS Tier 0 validators)
## Based On
- text_toolz library patterns
- extractor pipeline s02 normalization
- NFKC Unicode standard
عرض على GitHub