huggingface-tokenizers

# HuggingFace Tokenizers - Fast Tokenization for NLP Fast, production-ready tokenizers with Rust performance and Python ease-of-use. ## When to use HuggingFace Tokenizers **Use HuggingFace Tokenizers when:** - Need extremely fast tokenization (<20s per GB of text) - Training custom tokenizers from scratch - Want alignment tracking (token → original text position) - Building production NLP pipelines - Need to tokenize large corpora efficiently **Performance**: - **Speed**: <20 seconds to tokenize 1GB on CPU - **Implementation**: Rust core with Python/Node.js bindings - **Efficiency**: 10-100× faster than pure Python implementations **Use alternatives instead**: - **SentencePiece**: Language-independent, used by T5/ALBERT - **tiktoken**: OpenAI's BPE tokenizer for GPT models - **transformers AutoTokenizer**: Loading pretrained only (uses this library internally) ## Quick start ### Installation ```bash # Install tokenizers pip install tokenizers # With transformers integration pip install tokenizers transformers ``` ### Load pretrained tokenizer ```python from tokenizers import Tokenizer # Load from HuggingFace Hub tokenizer = Tokenizer.from_pretrained("bert-base-uncased") # Encode text output = tokenizer.encode("Hello, how are you?") print(output.tokens) # ['hello', ',', 'how', 'are', 'you', '?'] print(output.ids) # [7592, 1010, 2129, 2024, 2017, 1029] # Decode back text = tokenizer.decode(output.ids) print(text) # "hello, how are you?" ``` ### Train custom BPE tokenizer ```python from tokenizers import Tokenizer from tokenizers.models import BPE from tokenizers.trainers import BpeTrainer from tokenizers.pre_tokenizers import Whitespace # Initialize tokenizer with BPE model tokenizer = Tokenizer(BPE(unk_token="[UNK]")) tokenizer.pre_tokenizer = Whitespace() # Configure trainer trainer = BpeTrainer( vocab_size=30000, special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"], min_frequency=2 ) # Train on files files = ["train.txt", "validation.txt"] tokenizer.train(files, trainer) # Save tokenizer.save("my-tokenizer.json") ``` **Training time**: ~1-2 minutes for 100MB corpus, ~10-20 minutes for 1GB ### Batch encoding with padding ```python # Enable padding tokenizer.enable_padding(pad_id=3, pad_token="[PAD]") # Encode batch texts = ["Hello world", "This is a longer sentence"] encodings = tokenizer.encode_batch(texts) for encoding in encodings: print(encoding.ids) # [101, 7592, 2088, 102, 3, 3, 3] # [101, 2023, 2003, 1037, 2936, 6251, 102] ``` ## Tokenization algorithms ### BPE (Byte-Pair Encoding) **How it works**: 1. Start with character-level vocabulary 2. Find most frequent character pair 3. Merge into new token, add to vocabulary 4. Repeat until vocabulary size reached **Used by**: GPT-2, GPT-3, RoBERTa, BART, DeBERTa ```python from tokenizers import Tokenizer from tokenizers.models import BPE from tokenizers.trainers import BpeTrainer from tokenizers.pre_tokenizers import ByteLevel tokenizer = Tokenizer(BPE(unk_token="<|endoftext|>")) tokenizer.pre_tokenizer = ByteLevel() trainer = BpeTrainer( vocab_size=50257, special_tokens=["<|endoftext|>"], min_frequency=2 ) tokenizer.train(files=["data.txt"], trainer=trainer) ``` **Advantages**: - Handles OOV words well (breaks into subwords) - Flexible vocabulary size - Good for morphologically rich languages **Trade-offs**: - Tokenization depends on merge order - May split common words unexpectedly ### WordPiece **How it works**: 1. Start with character vocabulary 2. Score merge pairs: `frequency(pair) / (frequency(first) × frequency(second))` 3. Merge highest scoring pair 4. Repeat until vocabulary size reached **Used by**: BERT, DistilBERT, MobileBERT ```python from tokenizers import Tokenizer from tokenizers.models import WordPiece from tokenizers.trainers import WordPieceTrainer from tokenizers.pre_tokenizers import Whitespace from tokenizers.normalizers import BertNormalizer tokenizer = Tokenizer(WordPiece(unk_token="[UNK]")) tokenizer.normalizer = BertNormalizer(lowercase=True) tokenizer.pre_tokenizer = Whitespace() trainer = WordPieceTrainer( vocab_size=30522, special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"], continuing_subword_prefix="##" ) tokenizer.train(files=["corpus.txt"], trainer=trainer) ``` **Advantages**: - Prioritizes meaningful merges (high score = semantically related) - Used successfully in BERT (state-of-the-art results) **Trade-offs**: - Unknown words become `[UNK]` if no subword match - Saves vocabulary, not merge rules (larger files) ### Unigram **How it works**: 1. Start with large vocabulary (all substrings) 2. Compute loss for corpus with current vocabulary 3. Remove tokens with minimal impact on loss 4. Repeat until vocabulary size reached **Used by**: ALBERT, T5, mBART, XLNet (via SentencePiece) ```python from tokenizers import Tokenizer from tokenizers.models import Unigram from tokenizers.trainers import UnigramTrainer tokenizer = Tokenizer(Unigram()) trainer = UnigramTrainer( vocab_size=8000, special_tokens=["<unk>", "<s>", "</s>"], unk_token="<unk>" ) tokenizer.train(files=["data.txt"], trainer=trainer) ``` **Advantages**: - Probabilistic (finds most likely tokenization) - Works well for languages without word boundaries - Handles diverse linguistic contexts **Trade-offs**: - Computationally expensive to train - More hyperparameters to tune ## Tokenization pipeline Complete pipeline: **Normalization → Pre-tokenization → Model → Post-processing** ### Normalization Clean and standardize text: ```python from tokenizers.normalizers import NFD, StripAccents, Lowercase, Sequence tokenizer.normalizer = Sequence([ NFD(), # Unicode normalization (decompose) Lowercase(), # Convert to lowercase StripAccents() # Remove accents ]) # Input: "Héllo WORLD" # After normalization: "hello world" ``` **Common normalizers**: - `NFD`, `NFC`, `NFKD`, `NFKC` - Unicode normalization forms - `Lowercase()` - Convert to lowercase - `StripAccents()` - Remove accents (é → e) - `Strip()` - Remove whitespace - `Replace(pattern, content)` - Regex replacement ### Pre-tokenization Split text into word-like units: ```python from tokenizers.pre_tokenizers import Whitespace, Punctuation, Sequence, ByteLevel # Split on whitespace and punctuation tokenizer.pre_tokenizer = Sequence([ Whitespace(), Punctuation() ]) # Input: "Hello, world!" # After pre-tokenization: ["Hello", ",", "world", "!"] ``` **Common pre-tokenizers**: - `Whitespace()` - Split on spaces, tabs, newlines - `ByteLevel()` - GPT-2 style byte-level splitting - `Punctuation()` - Isolate punctuation - `Digits(individual_digits=True)` - Split digits individually - `Metaspace()` - Replace spaces with ▁ (SentencePiece style) ### Post-processing Add special tokens for model input: ```python from tokenizers.processors import TemplateProcessing # BERT-style: [CLS] sentence [SEP] tokenizer.post_processor = TemplateProcessing( single="[CLS] $A [SEP]", pair="[CLS] $A [SEP] $B [SEP]", special_tokens=[ ("[CLS]", 1), ("[SEP]", 2), ], ) ``` **Common patterns**: ```python # GPT-2: sentence <|endoftext|> TemplateProcessing( single="$A <|endoftext|>", special_tokens=[("<|endoftext|>", 50256)] ) # RoBERTa: <s> sentence </s> TemplateProcessing( single="<s> $A </s>", pair="<s> $A </s> </s> $B </s>", special_tokens=[("<s>", 0), ("</s>", 2)] ) ``` ## Alignment tracking Track token positions in original text: ```python output = tokenizer.encode("Hello, world!") # Get token offsets for token, offset in zip(output.tokens, output.offsets): start, end = offset print(f"{token:10} → [{start:2}, {end:2}): {text[start:end]!r}") # Output: # hello → [ 0, 5): 'Hello' # , → [ 5, 6): ',' # world → [ 7, 12): 'world' # ! → [12, 13): '!' ``` **Use cases**: - Named entity recognition (map predictions back to text) - Question answering (extract answer spans) - Token classification (align labels to original positions) ## Integration with transformers ### Load with AutoTokenizer ```python from transformers import AutoTokenizer # AutoTokenizer automatically uses fast tokenizers tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased") # Check if using fast tokenizer print(tokenizer.is_fast) # True # Access underlying tokenizers.Tokenizer fast_tokenizer = tokenizer.backend_tokenizer print(type(fast_tokenizer)) # <class 'tokenizers.Tokenizer'> ``` ### Convert custom tokenizer to transformers ```python from tokenizers import Tokenizer from transformers import PreTrainedTokenizerFast # Train custom tokenizer tokenizer = Tokenizer(BPE()) # ... train tokenizer ... tokenizer.save("my-tokenizer.json") # Wrap for transformers transformers_tokenizer = PreTrainedTokenizerFast( tokenizer_file="my-tokenizer.json", unk_token="[UNK]", pad_token="[PAD]", cls_token="[CLS]", sep_token="[SEP]", mask_token="[MASK]" ) # Use like any transformers tokenizer outputs = transformers_tokenizer( "Hello world", padding=True, truncation=True, max_length=512, return_tensors="pt" ) ``` ## Common patterns ### Train from iterator (large datasets) ```python from datasets import load_dataset # Load dataset dataset = load_dataset("wikitext", "wikitext-103-raw-v1", split="train") # Create batch iterator def batch_iterator(batch_size=1000): for i in range(0, len(dataset), batch_size): yield dataset[i:i + batch_size]["text"] # Train tokenizer tokenizer.train_from_iterator( batch_iterator(), trainer=trainer, length=len(dataset) # For progress bar ) ``` **Performance**: Processes 1GB in ~10-20 minutes ### Enable truncation and padding ```python # Enable truncation tokenizer.enable_truncation(max_length=512) # Enable padding tokenizer.enable_padding( pad_id=tokenizer.token_to_id("[PAD]"), pad_token="[PAD]", length=512 # Fixed length, or None for batch max ) # Encode with both output = tokenizer.encode("This is a long sentence that will be truncated...") print(len(output.ids)) # 512 ``` ### Multi-processing ```python from tokenizers import Tokenizer from multiprocessing import Pool # Load tokenizer tokenizer = Tokenizer.from_file("tokenizer.json") def encode_batch(texts): return tokenizer.encode_batch(texts) # Process large corpus in parallel with Pool(8) as pool: # Split corpus into chunks chunk_size = 1000 chunks = [corpus[i:i+chunk_size] for i in range(0, len(corpus), chunk_size)] # Encode in parallel results = pool.map(encode_batch, chunks) ``` **Speedup**: 5-8× with 8 cores ## Performance benchmarks ### Training speed | Corpus Size | BPE (30k vocab) | WordPiece (30k) | Unigram (8k) | |-------------|-----------------|-----------------|--------------|