unicode-text-correctness

name	unicode-text-correctness
description	Implements and fixes correct text/Unicode handling — pinning UTF-8 end-to-end, detecting BOM/legacy charsets, NFC/NFD normalization, grapheme-aware length/slicing/truncation/reversal, locale-aware collation and full case-folding, and homoglyph/confusable/bidi spoofing defenses.
when_to_use	Code measures, slices, truncates, reverses, sorts, lowercases, or compares strings containing emoji, combining marks, or CJK; or bugs show mojibake, emoji counting as length 4, truncation splitting a character, equal-looking usernames comparing unequal, broken accented sorting, or double-encoding. Distinct from regex-build (pattern matching) and validate-data-quality (column-level rules, not character semantics).

When to Use

Reach for this when the bug is about what a character is — its bytes, boundaries, identity, or order — not about pattern matching or business rules:

• "Emoji 👨‍👩‍👧 counts as length 7 / truncates to a broken � / reverses into garbage"
• "Twitter-style 120 chars limit cuts a flag emoji or é in half"
• "Two usernames look identical but == says they differ" (or the reverse: a spoof passes)
• "Accented words sort after z / ä doesn't sort near a"
• "Text came in as é / ’ / é — mojibake / double-encoding"
• ".toLowerCase() breaks Turkish İ, German ß, or fails to match İstanbul"
• "MySQL stores emoji as ???? / IDN domain аpple.com (Cyrillic а) phishes users"

NOT this skill:

• Writing/debugging a regex pattern (email/slug/\d over-matching) → regex-build
• Column-level assertions (no nulls/dupes, value ranges, freshness) → validate-data-quality
• Schema/charset migration mechanics (lock contention, rollback of an ALTER) → db-migration-safety
• Whether a confusable username is an actual attack you must report in a diff → security-review (this skill builds the defense; security-review audits for its absence)

Steps

1. Know the four length units — pick one deliberately, never let the language pick for you. Most "Unicode bugs" are using the wrong unit.

   Unit	What it counts	"é" (NFD)	"👨‍👩‍👧"	Use for
   Bytes	UTF-8 octets	3	18	storage size, network frames, DB byte limits
   Code units	UTF-16 slots (JS .length, Java char)	2	7	almost never — this is the trap
   Code points	Unicode scalars	2	5	normalization input, codepoint ranges
   Grapheme clusters	user-perceived characters	1	1	length shown to users, truncation, cursor, slicing

   Default for any user-facing length, limit, slice, or reverse: grapheme clusters. JS "👨‍👩‍👧".length === 7 and [..."👨‍👩‍👧"].length === 5 are both wrong for "how many characters"; only a segmenter gives 1.

2. Count and slice on grapheme boundaries — use a real segmenter, do not split on code points. Built-ins:

   const seg = new Intl.Segmenter(undefined, { granularity: "grapheme" });
   const graphemes = [...seg.segment(s)].map(x => x.segment);
   const len = graphemes.length;                 // user-visible length
   const head = graphemes.slice(, ).(); 
    reversed = graphemes.().(); 
   

Common Errors

• Using .length (JS/Java UTF-16) as character count. Counts code units → emoji = 2–7, BMP CJK = 1. Fix: Intl.Segmenter graphemes for user counts.
• Splitting on code points and calling it grapheme-safe. [...str] keeps é(NFC) whole but shatters 👨‍👩‍👧 (5 codepoints) and a base+combining e+◌́. Fix: segment graphemes, not codepoints.
• Byte-cap truncation (s[:200], substr). Cuts mid-codepoint → �, or splits a base from its combining mark / a ZWJ sequence. Fix: trim whole graphemes until under the byte cap.
• Comparing/indexing without normalizing. NFC café ≠ NFD café; one inserts, the other duplicates past a UNIQUE constraint. Fix: NFC both sides before ==, hash, and the DB write.
• toLowerCase() for identity/security checks. Misses ß/ss, breaks Turkish İ/ı, locale-dependent. Fix: full case-fold (casefold()), NFC first.
• Sorting by codepoint/byte. Z before a, accents dumped after z, wrong per language. Fix: ICU/CLDR collator with an explicit locale.
• MySQL utf8 (3-byte alias). Silently stores emoji/4-byte chars as ???? or errors. Fix: utf8mb4 everywhere — column, table, connection.
• Double-decoding / re-encoding. Decoding an already-str value (or treating UTF-8 bytes as Latin-1 then re-encoding) → é, ’. Fix: decode exactly once at the boundary; keep Unicode internally.
• Not stripping the BOM. Leading U+FEFF breaks JSON.parse, makes the first CSV column key invisible. Fix: strip a leading  on read.
• Reversing a string by codepoint/char. Scrambles emoji ZWJ sequences and detaches combining marks (á → ́a). Fix: reverse grapheme clusters.
• NFKC on display text. Lossy: ²→, →, full-width collapses. Fix: NFKC only for fold-keys/identifiers; store NFC for display.

Verify

Test every text op against a fixed adversarial corpus — at minimum: "á" (e + combining acute, NFD á), "á" (NFC á), "👨‍👩‍👧‍👦" (ZWJ family), "🇯🇵" (regional-indicator flag), "ẹ́" (stacked combining marks), "한국어" (Hangul), "Ｈｅｌｌｏ" (full-width), "раypal" (mixed-script Cyrillic), "safe‮txt.exe" (bidi override), "hi" (BOM), "café" in NFC and NFD.

1. Grapheme length: the ZWJ family and a flag emoji each report length 1; "á" reports 1. Not 2, 4, or 7.
2. Truncation: truncating the corpus to N graphemes never yields a �, never splits a ZWJ sequence, and never strands a combining mark; utf8Bytes(result) <= byteCap when a byte cap applies.
3. Reverse: reversing "👨‍👩‍👧" returns it unchanged (single grapheme); reversing "áb" keeps á intact.
4. Normalization equality: NFC "café" and NFD "café" compare equal and hash equal after .normalize("NFC"); inserting both into a table with a UNIQUE(NFC) key yields one row.
5. Case-fold: "ß" matches "SS"/"ss" under full case-fold; "İstanbul" matches per Turkish locale and is not silently mangled in the default locale.
6. Collation: sorting ["z","ä","a","Z"] under de collator puts ä adjacent to a and is not codepoint order (Z before a).
7. Confusable/bidi: "раypal" is flagged confusable with an existing "paypal" and mixed-script-rejected; the bidi-override string is rejected or its overrides stripped before storage/display.
8. Round-trip: a string written to the DB (utf8mb4) and read back is byte-identical including emoji; a BOM-prefixed file parses with no phantom first key; an IDN host round-trips through Punycode and back.

Done = grapheme-unit length/slice/truncate/reverse are all correct on the ZWJ + flag + combining-mark corpus, NFC-normalized values compare/hash/dedup equal across forms, case-insensitive matching uses full case-folding and sorting uses a locale collator, confusable + mixed-script + bidi spoofs are rejected, and emoji round-trip cleanly through the utf8mb4 store with no ????/�/mojibake.