菜单
Implements and fixes correct text/Unicode handling — pinning UTF-8 end-to-end, detecting BOM/legacy charsets, NFC/NFD normalization, grapheme-aware length/slicing/truncation/reversal, locale-aware collation and full case-folding, and homoglyph/confusable/bidi spoofing defenses.
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
基于 SOC 职业分类
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/Sir-chawakorn/sanook-cli --skill unicode-text-correctness命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
正在显示 SKILL.md
Gets machine-parseable JSON out of an LLM reliably — prefer provider-enforced grammar (OpenAI `response_format:{type:"json_schema",strict:true}`, Anthropic tool-calling / `tool_choice:{type:"tool"}`, Gemini `responseSchema`) over a "respond in JSON" prompt, define the contract once as a Pydantic/Zod model and validate every response (never trust the string), use constrained decoding / grammars for open weights (Outlines, llguidance, llama.cpp GBNF, vLLM `guided_json`), keep schemas SHALLOW with required fields + enums + `additionalProperties:false`, and build a repair-and-retry loop that handles refusals, `length`/`max_tokens` truncation, and streaming partial JSON (`partial-json-parser`). Schema-guided generation eliminates parse-error retries; validate-and-repair is the backstop, not the primary mechanism.
Builds production data grids that stay fast and accessible at 10k–1M+ rows — decide server-side vs client-side sort/filter/paginate by the dataset-fits-in-memory test (client only under ~10k rows, otherwise push to the API and treat the table as a controlled view of server state), ROW-VIRTUALIZE with TanStack Virtual or react-window so only the visible window mounts (fixed estimateSize, overscan 5–10, measureElement for dynamic rows, contain:strict, and a real scroll container — never table-layout:auto over thousands of rows), build the headless logic with TanStack Table v8 (or AG Grid when you need pinning/grouping/enterprise out of the box), add column resize/reorder/pin, inline edit with optimistic update + rollback on error, row selection with a stable rowId, full keyboard nav with roving tabindex over an ARIA grid (role=grid/row/gridcell, aria-sort, aria-rowcount/aria-rowindex so virtualization stays announced), position:sticky headers, streaming CSV export that doesn't block the main thread, and explici
Builds semantic/vector search — pick an embedding model + dimensionality (and whether to truncate Matryoshka dims) and the matching distance metric (cosine/dot/L2, normalize to unit length so cosine == dot and IP is correct), an ANN index with the recall/latency/memory tradeoff understood (HNSW M/efConstruction/efSearch for low-latency RAM-resident; IVF-PQ nlist/nprobe/PQ for billion-scale compressed; flat/exact for <100k) in pgvector/Qdrant/Milvus/FAISS/Pinecone, chunking + overlap + per-chunk metadata for filtering, HYBRID retrieval fusing BM25 + dense by Reciprocal Rank Fusion (RRF, k≈60) not score addition, a cross-encoder/Cohere reranker over the top-50→k, correct pre-filter-vs-ANN interaction (filterable HNSW, not post-filter that starves k), and offline eval with recall@k / nDCG@10 / MRR against a labeled qrels set. Quantize (scalar/PQ) only after measuring recall loss; tune efSearch/nprobe to a recall target, not a guess.
| name | unicode-text-correctness |
| description | Implements and fixes correct text/Unicode handling — pinning UTF-8 end-to-end, detecting BOM/legacy charsets, NFC/NFD normalization, grapheme-aware length/slicing/truncation/reversal, locale-aware collation and full case-folding, and homoglyph/confusable/bidi spoofing defenses. |
| when_to_use | Code measures, slices, truncates, reverses, sorts, lowercases, or compares strings containing emoji, combining marks, or CJK; or bugs show mojibake, emoji counting as length 4, truncation splitting a character, equal-looking usernames comparing unequal, broken accented sorting, or double-encoding. Distinct from regex-build (pattern matching) and validate-data-quality (column-level rules, not character semantics). |
Reach for this when the bug is about what a character is — its bytes, boundaries, identity, or order — not about pattern matching or business rules:
👨👩👧 counts as length 7 / truncates to a broken � / reverses into garbage"120 chars limit cuts a flag emoji or é in half"== says they differ" (or the reverse: a spoof passes)z / ä doesn't sort near a"é / ’ / é — mojibake / double-encoding".toLowerCase() breaks Turkish İ, German ß, or fails to match İstanbul"???? / IDN domain аpple.com (Cyrillic а) phishes users"NOT this skill:
\d over-matching) → regex-buildALTER) → db-migration-safetyKnow the four length units — pick one deliberately, never let the language pick for you. Most "Unicode bugs" are using the wrong unit.
| Unit | What it counts | "é" (NFD) | "👨👩👧" | Use for |
|---|---|---|---|---|
| Bytes | UTF-8 octets | 3 | 18 | storage size, network frames, DB byte limits |
| Code units | UTF-16 slots (JS .length, Java char) | 2 | 7 | almost never — this is the trap |
| Code points | Unicode scalars | 2 | 5 | normalization input, codepoint ranges |
| Grapheme clusters | user-perceived characters | 1 | 1 | length shown to users, truncation, cursor, slicing |
Default for any user-facing length, limit, slice, or reverse: grapheme clusters. JS "👨👩👧".length === 7 and [..."👨👩👧"].length === 5 are both wrong for "how many characters"; only a segmenter gives 1.
Count and slice on grapheme boundaries — use a real segmenter, do not split on code points. Built-ins:
const seg = new Intl.Segmenter(undefined, { granularity: "grapheme" });
const graphemes = [...seg.segment(s)].map(x => x.segment);
const len = graphemes.length; // user-visible length
const head = graphemes.slice(, ).();
reversed = graphemes.().();
.length (JS/Java UTF-16) as character count. Counts code units → emoji = 2–7, BMP CJK = 1. Fix: Intl.Segmenter graphemes for user counts.[...str] keeps é(NFC) whole but shatters 👨👩👧 (5 codepoints) and a base+combining e+◌́. Fix: segment graphemes, not codepoints.s[:200], substr). Cuts mid-codepoint → �, or splits a base from its combining mark / a ZWJ sequence. Fix: trim whole graphemes until under the byte cap.café ≠ NFD café; one inserts, the other duplicates past a UNIQUE constraint. Fix: NFC both sides before ==, hash, and the DB write.toLowerCase() for identity/security checks. Misses ß/ss, breaks Turkish İ/ı, locale-dependent. Fix: full case-fold (casefold()), NFC first.Z before a, accents dumped after z, wrong per language. Fix: ICU/CLDR collator with an explicit locale.utf8 (3-byte alias). Silently stores emoji/4-byte chars as ???? or errors. Fix: utf8mb4 everywhere — column, table, connection.str value (or treating UTF-8 bytes as Latin-1 then re-encoding) → é, ’. Fix: decode exactly once at the boundary; keep Unicode internally.U+FEFF breaks JSON.parse, makes the first CSV column key invisible. Fix: strip a leading on read.á → ́a). Fix: reverse grapheme clusters.²→, →, full-width collapses. Fix: NFKC only for fold-keys/identifiers; store NFC for display.Test every text op against a fixed adversarial corpus — at minimum: "á" (e + combining acute, NFD á), "á" (NFC á), "👨👩👧👦" (ZWJ family), "🇯🇵" (regional-indicator flag), "ẹ́" (stacked combining marks), "한국어" (Hangul), "Ｈｅｌｌｏ" (full-width), "раypal" (mixed-script Cyrillic), "safetxt.exe" (bidi override), "hi" (BOM), "café" in NFC and NFD.
"á" reports 1. Not 2, 4, or 7.�, never splits a ZWJ sequence, and never strands a combining mark; utf8Bytes(result) <= byteCap when a byte cap applies."👨👩👧" returns it unchanged (single grapheme); reversing "áb" keeps á intact."café" and NFD "café" compare equal and hash equal after .normalize("NFC"); inserting both into a table with a UNIQUE(NFC) key yields one row."ß" matches "SS"/"ss" under full case-fold; "İstanbul" matches per Turkish locale and is not silently mangled in the default locale.["z","ä","a","Z"] under de collator puts ä adjacent to a and is not codepoint order (Z before a)."раypal" is flagged confusable with an existing "paypal" and mixed-script-rejected; the bidi-override string is rejected or its overrides stripped before storage/display.utf8mb4) and read back is byte-identical including emoji; a BOM-prefixed file parses with no phantom first key; an IDN host round-trips through Punycode and back.Done = grapheme-unit length/slice/truncate/reverse are all correct on the ZWJ + flag + combining-mark corpus, NFC-normalized values compare/hash/dedup equal across forms, case-insensitive matching uses full case-folding and sorting uses a locale collator, confusable + mixed-script + bidi spoofs are rejected, and emoji round-trip cleanly through the utf8mb4 store with no ????/�/mojibake.
regex module \X (regex.findall(r"\X", s)) — stdlib re/len() give code points, not graphemes.unicode-segmentation .graphemes(true). Go: rivo/uniseg. Swift: String is already grapheme-correct (.count).VARCHAR(n) is bytes) also applies, trim graphemes until utf8Bytes(result) <= cap — never cut at byte n directly.Normalize to NFC at every boundary you store, compare, hash, or index. "é" has two encodings (NFC U+00E9 = 1 codepoint; NFD U+0065 U+0301 = 2). They render identically but are != and hash differently. Rule:
UNIQUE index only on NFC strings — never store one form and look up another (macOS filesystem returns NFD; HTTP/most input is NFC → a path from disk won't match a stored key without normalizing both sides).const key = s.normalize("NFC"); // JS
import unicodedata; key = unicodedata.normalize("NFC", s) # Python
Use NFKC (compatibility) only for identifiers/search keys where you want ①→1, fi→fi, full-width Ａ→A folded together — it is lossy, so never NFKC user display text.
Compare case-insensitively with full case-folding, not lower(); sort with a locale collator, not byte order.
str.casefold() (Python) / String::to_lowercase (Rust) is the floor — .toLowerCase()/.toUpperCase() is not enough. "ß".casefold() == "ss"; Turkish "İ" vs "i" differ by locale. Never use .toLowerCase() for a security or identity comparison — NFC then fold both sides: compare a.normalize("NFC") folded vs b.normalize("NFC") folded.Z(0x5A) before a(0x61) and accented letters after z. Use an ICU/CLDR collator: new Intl.Collator("de", { sensitivity: "base" }).compare(a, b) (JS), PyICU.Collator or locale.strxfrm (Python), COLLATE "de-x-icu" (Postgres). Pin the locale explicitly — the "right" order for ä/ö differs by language (German vs Swedish).Defend identifiers (usernames, domains, package names) against confusables and mixed-script spoofing. Equal-looking must mean equal-compared, and visually-deceptive must be rejected:
раypal→paypal) via the Unicode confusables table (confusable_homoglyphs, ICU usprep, unicode-security crate) and compare skeletons against existing identifiers.аpple mixing Cyrillic а + Latin); permit only known-safe combos (Latin+Han+Hiragana for JP). Reject whole-script confusables (all-Cyrillic аррӏе).U+202A–202E, U+2066–2069, and zero-width U+200B/200C/200D/FEFF in identifiers and filenames — safe.txtgpj.exe displays as safe.txtexe.jpg (Trojan Source). NFKC-fold identifiers before storing.Pin UTF-8 across storage and transport — no implicit charset anywhere.
utf8mb4 (the 3-byte utf8 alias silently drops emoji → ????); set table and connection charset + a _unicode_ci/utf8mb4_0900_ai_ci collation. Postgres: ENCODING 'UTF8' + ICU collation per UTF-8 column.Content-Type: …; charset=utf-8; read charset from the response header, fall back to BOM, then to the declared meta — never assume Latin-1.U+FEFF on read (it corrupts JSON parse and the first field of CSV); do not emit a BOM in UTF-8 output unless a consumer (Excel CSV) demands it.chardet/charset-normalizer/ICU CharsetDetector, decode once to Unicode, then work in UTF-8 — and never re-decode an already-decoded string (the cause of ’ double-encoding mojibake).xn--…) for transport, but display the Unicode form only after the confusable check in step 5.Lock the behavior with adversarial test strings (see Verify) before declaring text handling correct.
2fifi