| name | script-aware-romanization-pipeline |
| description | Adding a "show pronunciation" or transliteration feature across many non-Latin scripts in a Kotlin Multiplatform module — without pulling in ICU — needs a per-script dispatch built on Unicode block ranges rather than a locale, one line at a time rather than one song at a time, and a hard line between scripts that reduce to a table and the one or two that need a real dictionary. Use when a transliteration result is guessed for the wrong language, an unsupported platform fails outright instead of falling back, or a line that mixes two scripts (an original lyric plus an English aside) picks the wrong one. |
A script-aware romanization pipeline
One entry point takes a line of text and a set of enabled languages, and returns a Latin-script
reading of that line or null:
fun romanize(line: String, enabled: Set<RomanizationLanguage>): String?
Everything interesting is in how it decides which of several very different transliteration
strategies to run, per line, not per song: a lyric sheet routinely interleaves an original-script
line with a plain-English one, and romanizing the English half produces nonsense. Detection therefore
runs fresh on every line, and the twelve supported languages split cleanly into two families —
arithmetic/table scripts that need nothing beyond commonMain, and two scripts (Japanese, Chinese)
whose readings genuinely depend on surrounding words and therefore need a real dictionary behind an
expect/actual boundary. Getting that boundary as narrow as possible is most of the design.
Traps
Detecting script by Unicode code-block ranges works — but only in a specific order, not any
order. A block-range test (char in ''..'ヿ' for kana, and so on for Han, Hangul,
Devanagari, Gurmukhi, Cyrillic) needs no ICU and no locale, but a Japanese line is written mostly in
kanji with only a few kana particles — counting characters and taking a majority calls it Chinese.
The fix is a priority order, not a count: kana anywhere at all proves the line is Japanese (Chinese
text contains none), so the kana check must run, and win, before the Han check. Any per-script
detector built from "which characters are present" needs this kind of precedence rule wherever two
scripts' ranges can co-occur in the same line.
One Unicode block can hold several languages that romanize differently, and the block alone cannot
tell them apart. Seven Cyrillic-alphabet languages here share one range. Resolving which one a
line is in has to come from letters each language has that the others do not — ordered
most-distinctive first: two languages that share almost their whole alphabet except for one
language's few unique letters must have those unique letters checked before the shared ones, or the
shared letters match first and the unique ones never get a chance. Skip the disambiguation step
entirely when only one candidate language is enabled — there is nothing to guess, and running a
letter-frequency heuristic on a known-single-language line only risks getting it wrong for no reason.
When nothing distinctive matches, fall back to a single hardcoded default, not a computed "closest
alphabet" comparison: the source reaches for Russian specifically — its own comment calls it the most
common of the seven and the one carrying none of the others' unique letters — and only when Russian
itself is not enabled does it fall through to cyrillicEnabled.first(), whichever language that
happens to be in iteration order, with no comparison run at all (RomanizationLanguage.kt:71-73, 99).
A script that looks alphabetic can still be an abugida, and a lookup table alone cannot transliterate
one. Devanagari and Gurmukhi consonants carry an inherent vowel that a following mark can replace or
delete — a plain Map<Char, String> gets the base consonant right and the vowel wrong on every
syllable that isn't the default case. This needs a small state machine reading one character of
lookahead, not a bigger map; see combining-chars-break-char-literals for the specific complication
this creates once one of those marks is a combining character rather than its own letter.
Only make a script's transliteration depend on a real dictionary when the rule genuinely cannot be
local to one character. Two of twelve languages here need a platform library — Japanese, because a
kanji's reading depends on the word it's in, and Chinese, because some characters are heteronyms whose
pronunciation depends on the surrounding word. Every other script resolves per character or by pure
arithmetic and needs nothing beyond commonMain: a syllabic script's alphabet is a small map, and a
syllable-block script decomposes with integer arithmetic instead of a table at all
((codepoint - base) / consonantCount / vowelCount, or similar, covers the whole script from three
short tables). Reach for a dictionary only for cross-word disambiguation and morphology — properties a
per-character table cannot express by construction. Even where a dictionary IS used, a heteronym
looked up one character at a time returns several valid readings with no ranking between them —
taking the first is a guess; the honest fix is a word-aware lookup, not tuning which array index gets
picked.
A dictionary-backed language can still be a two-stage pipeline where only the first stage needs the
dictionary. Getting from kanji to a phonetic reading needs a real morphological analyzer; getting
from that reading to a Latin-script one is still pure table lookup plus a little lookahead — so it
belongs in shared, dictionary-free code, testable without the platform dependency the first stage
needs. Collapsing both into "the platform-specific implementation" denies every other platform a stage
it could have run itself.
Inside that shared table-lookup stage, a multi-character token must be tried before its own prefix
is tried alone. A syllable spelled with two characters that together make one sound is a single
token in the output table; matching one character at a time first consumes the first of the pair as
its own (wrong) sound and leaves the second stranded. Check known multi-character sequences starting
at the current position before falling back to the single-character table, every time the read
position advances — the shorter match is not a safe default merely because it comes first in a naive
loop.
Every reason a romanizer might have "nothing to add" should collapse to one signal, not several.
A line already in Latin, a language switched off in settings, a platform with no library for that
script, and a transliteration that happens to equal its own input are four situations with the same
correct behavior: show nothing extra. Returning null for all four — rather than throwing for the
unsupported-platform case — lets every caller be one if (result != null) instead of a when over
failure reasons, and lets a platform ship with a script's dictionary simply absent, not stubbed.
"The dictionary is not downloaded yet" and "this platform has no dictionary at all" should be the
same return value, not two. A library that ships its dictionary as an on-demand download has a
window where the feature is temporarily unavailable rather than permanently unavailable — resist the
urge to add a third state for it. Returning the same null for "not ready yet" as for "not supported
here" means every caller that already handles the permanent case handles the temporary one for free,
with no extra branch anywhere — see on-demand-dictionary-asset for the provider side of this same
null, where the download-not-finished check actually lives.
Two consumers that both need to answer "is this character script X" must call the same function, not
duplicate the range check. A script detector and a per-character transliterator over the same script
are tempting to write as two separate range checks — one per change that touches either. Duplicated
tests drift the moment one is edited without the other; a single shared predicate cannot.
A specific third-party formatting flag combination throws instead of formatting. A romanization
library asked for both diacritic tone marks and a particular "how to spell a Latin letter with no
Latin equivalent" option throws a checked exception rather than silently producing something wrong —
worth knowing before it surfaces from production traffic instead of from reading the docs. Confirm the
exact incompatible pairing for your own formatting library before wiring up every option combination.
Verifying it
Run from the root of a Kotlin Multiplatform module with a similar split (adapt file names):
grep -n "expect object\|fun " service/lyricsService/src/commonMain/kotlin/org/simpmusic/lyrics/romanization/PlatformRomanizer.kt
grep -n "actual fun" service/lyricsService/src/iosMain/kotlin/org/simpmusic/lyrics/romanization/PlatformRomanizer.ios.kt
grep -n "fun detectScript" -A8 service/lyricsService/src/commonMain/kotlin/org/simpmusic/lyrics/romanization/RomanizationLanguage.kt
grep -n "size == 1" -A6 service/lyricsService/src/commonMain/kotlin/org/simpmusic/lyrics/romanization/RomanizationLanguage.kt
grep -n "val pair = \|val youon = \|val single = " service/lyricsService/src/commonMain/kotlin/org/simpmusic/lyrics/romanization/KanaRomanizer.kt
JAR=$(find ~/.gradle/caches -iname "pinyin4j-*.jar" ! -iname "*sources*" ! -iname "*javadoc*" | head -1)
javap -p -classpath "$JAR" net.sourceforge.pinyin4j.format.exception.BadHanyuPinyinOutputFormatCombination
Pass condition for 1: exactly two function signatures inside the expect object. For 2: both
actual fun bodies are a bare null, no imports of a native library. For 3: the kana check is the
first line in the function, before the Han check. For 4: a size == 1 early return precedes a list
literal whose comment says "most-distinctive first." For 5: pair and youon are computed and
checked before single is even read. For 6: javap prints the exception class — confirming it is a
real, named type the calling code can be shown to guard against, not a hypothetical.
