unicode-data

name	unicode-data
description	Regenerate the generated Unicode data modules (src/_grapheme_data.js, _general_data.js, _emoji_data.js, test/_unicode_testdata.js) and verify correctness. Use when bumping the Unicode version, editing scripts/unicode.js or scripts/lib/encoding.js, or changing grapheme break rules or the pair table.

Unicode data pipeline

node scripts/unicode.js downloads UCD files for the pinned UNICODE_VERSION (a const near the top of the script) into scripts/unicode_data/ (cached — delete a file to re-fetch) and regenerates:

• src/_grapheme_data.js — grapheme_data (encoded ranges), grapheme_cats (one base36 digit per range), grapheme_pairs (256-digit 16×16 break-decision table), GraphemeCategory enum
• src/_general_data.js, src/_emoji_data.js — flat membership tables
• test/_unicode_testdata.js — official GraphemeBreakTest cases consumed by yarn test

Never hand-edit generated files; fix the generator and re-run.

Encoding invariants (breaking any of these corrupts data silently)

• Ranges are delta + base36-VLQ encoded: 4 payload bits + 1 continuation bit per character, alphabet [0-9a-v], decoded by native parseInt(ch, 36) in src/core.js (decodeUnicodeData). Encoder lives in scripts/lib/encoding.js. base36 was chosen over base64 deliberately: no codec needed in the bundle, and it compresses better under gzip/brotli.
• Ranges must be sorted and non-overlapping (delta gaps are non-negative by construction).
• Flat tables pack end << 5 | category into Uint32 (findUnicodeRangeCategory unpacks with >>> 5 / & 31). Category max is 31; the largest real value is 23 (word break). A future category beyond 31 requires a format change in both core.js and the generator.
• Grapheme category 15 (InCB_Consonant) is internal-only — it exists so InCB=Consonant data folds into the main grapheme table instead of a separate module. It must never leak through the public API: graphemeSegments masks it to 0 in _catBegin/_catEnd.

Pair table semantics

grapheme_pairs[prevCat * 16 + curCat] values: 0 = break, 1 = no break, 2 = GB12/13 (Regional Indicator parity), 3 = GB11 (ExtPic + ZWJ sequence), 4 = GB9c (InCB linker). The stateful cases (2–4) are resolved by the packed sequence state in src/grapheme.js (nextState + the gate mask). Rule changes belong in buildGraphemePairTable() in scripts/unicode.js, not in hand edits to the emitted string.

Inlined fast paths

src/grapheme.js hardcodes hot regions (Latin, CJK, Hangul LV/LVT, PUA, variation selectors, E0000 tags, …) as computed checks plus three Uint8Array windows (T0: 0x0000–0x2FFF, T1: 0xA000–0xABFF, T2: 0x1F000–0x1FAFF); only the remainder reaches the binary-search tail. scripts/unicode.js asserts every inlined assumption against the raw UCD tables at codegen time, plus a tail-size cap (≤320 ranges). If regeneration throws an assertion, a new Unicode version changed a region that grapheme.js inlines — update the inline logic in src/grapheme.js to match reality; never weaken the assertion.

Verification checklist after any regen or rule change

1. yarn test — includes the official UCD GraphemeBreakTest suite, fast-check property tests against Intl.Segmenter (100k runs each), and the real-world corpus suite (test/corpora.js over committed test/_corpora/ fixtures — natural texts caught bugs both of the others missed; see issue #124).
2. For rule changes, deep-fuzz in the scratchpad: fast-check comparing graphemeSegments against Intl.Segmenter at ≥1M runs over full-Unicode strings. Divergences are not automatically bugs — the host ICU may lag or lead the pinned UNICODE_VERSION; adjudicate against the UAX #29 rules for the pinned version before "fixing" anything.
3. yarn bundle-stats:grapheme — data changes move bundle size; report the delta.
4. Add a changeset (yarn changeset) for anything user-visible; Unicode version bumps are user-visible.

On a Unicode version bump, also update UNICODE_VERSION_STRING in scripts/corpora.js (it pins the emoji-test.txt URL) and re-run node scripts/corpora.js to refresh the emoji corpus fixture. Downloads cache under scripts/corpora_data/ (gitignored); the fixtures in test/_corpora/ are committed.