sentencepiece

name	sentencepiece
description	Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.
category	llm-tools
version	1.0.0
author	Synthetic Sciences
license	MIT
tags	["Tokenization","SentencePiece","Language-Independent","BPE","Unigram","Multilingual","CJK Languages","Unicode","Deterministic","Google"]
dependencies	["sentencepiece","transformers"]

SentencePiece - Language-Independent Tokenization

Unsupervised tokenizer that works on raw text without language-specific preprocessing.

When to use SentencePiece

Use SentencePiece when:

• Building multilingual models (no language-specific rules)
• Working with CJK languages (Chinese, Japanese, Korean)
• Need reproducible tokenization (deterministic vocabulary)
• Want to train on raw text (no pre-tokenization needed)
• Require lightweight deployment (6MB memory, 50k sentences/sec)

Performance:

• Speed: 50,000 sentences/sec
• Memory: ~6MB for loaded model
• Languages: All (language-independent)

Use alternatives instead:

• HuggingFace Tokenizers: Faster training, more flexibility
• tiktoken: OpenAI models (GPT-3.5/4)
• BERT WordPiece: English-centric tasks

Quick start

Installation

# Python
pip install sentencepiece

# C++ (requires CMake)
git clone https://github.com/google/sentencepiece.git
cd sentencepiece
mkdir build && cd build
cmake .. && make -j $(nproc)
sudo make install

Train model

# Command-line (BPE with 8000 vocab)
spm_train --input=data.txt --model_prefix=m --vocab_size=8000 --model_type=bpe

# Python API
import sentencepiece as spm

spm.SentencePieceTrainer.train(
    input='data.txt',
    model_prefix='m',
    vocab_size=8000,
    model_type='bpe'
)

Training time: ~1-2 minutes for 100MB corpus

Encode and decode

import sentencepiece as spm

# Load model
sp = spm.SentencePieceProcessor(model_file='m.model')


pieces = sp.encode(, out_type=)
(pieces)  


ids = sp.encode(, out_type=)
(ids)  


text = sp.decode(ids)
(text)