sentencepiece
SentencePiece is Google's C++ subword tokenizer with Python bindings. You train a tokenizer directly on raw text using BPE or the unigram language model, get one self-contained .model file, and use it to encode text into pieces or ids and decode back losslessly, whitespace included. Because it treats input as a raw Unicode stream and marks spaces with a meta symbol, the same pipeline handles English, Chinese or Japanese with no language-specific pre-tokenizer. It is the tokenizer format behind T5, ALBERT and the early Llama generations.
Still the standard tool when you genuinely need to train your own tokenizer, and the 0.2.x wheels removed the old install pain. If you only consume pretrained models, you will almost never call it directly; it just rides along as a dependency.
Use it if
- You are training a model from scratch and need to build a tokenizer on your own corpus, especially multilingual or CJK text where there are no word boundaries to split on
- You need lossless round-tripping: decode(encode(text)) returns the exact original string, which word-based tokenizers that drop whitespace cannot guarantee
- Throughput matters: batch encoding runs multithreaded C++ and the project's own FLORES-200 benchmarks show it several times faster than Hugging Face fast tokenizers on raw multilingual text
- You want subword regularization or BPE-dropout, sampling different segmentations of the same text as data augmentation for translation and language models
- You are consuming a pretrained model: load that model's tokenizer through transformers AutoTokenizer instead of the raw .model file, because special-token conventions differ per model and are easy to get wrong by hand
- Your target is GPT-style byte-level BPE vocabularies: tiktoken and tokenizers speak those formats, sentencepiece cannot load them
- You want a pythonic API: this is a thin SWIG wrapper over C++, so error messages are terse, docstrings are minimal, and training is configured through dozens of string flags documented in repo markdown files
- You need to edit a trained vocabulary afterwards: the .model file is a protobuf that is effectively read-only unless you work directly with the model proto schema
Setup reality
pip install sentencepiece pulls a prebuilt wheel for every major platform, so the install itself is painless now. The friction is in training configuration: roughly forty flags (character_coverage, byte_fallback, user_defined_symbols, normalization_rule_name and friends) documented in doc/options.md rather than Python docstrings, so plan to read repo markdown. Training wants a text file on disk or a sentence iterator paired with a BytesIO model writer, and large corpora need input_sentence_size plus shuffle_input_sentence to keep memory in check. The wrapper answers to both CamelCase and snake_case method names, which is why tutorials never agree on spelling.
Patterns
Train a tokenizer from a raw text filetrain-model
import sentencepiece as spm
spm.SentencePieceTrainer.train(
input='corpus.txt',
model_prefix='m',
vocab_size=8000,
)
# writes m.model and m.vocabDefault model_type is 'unigram'. vocab_size that exceeds what the corpus can support raises an error; shrink it or add data.
Encode to pieces or ids and decode backencode-decode
import sentencepiece as spm
sp = spm.SentencePieceProcessor(model_file='m.model')
pieces = sp.encode('I saw a girl with a telescope.', out_type=str)
ids = sp.encode('I saw a girl with a telescope.', out_type=int)
sp.decode(ids) # exact original string
sp.decode(pieces) # samePieces carry a U+2581 meta symbol for spaces; detokenization is just join plus replacing that symbol, which is why round-trips are lossless.
Train a BPE model instead of unigramtrain-bpe-model
spm.SentencePieceTrainer.train(
input='corpus.txt',
model_prefix='bpe',
vocab_size=16000,
model_type='bpe',
)model_type accepts 'unigram', 'bpe', 'char' and 'word'; 'word' requires pre-tokenized input, which defeats most of the point.
Add BOS/EOS ids while encodingadd-bos-eos
ids = sp.encode('hello world', out_type=int, add_bos=True, add_eos=True)
sp.bos_id() # 1 by default
sp.eos_id() # 2 by default
sp.pad_id() # -1 unless set at training timepad is disabled by default; pass pad_id=3 (or similar) and pad_piece at training time if your framework needs a real pad token.
Sample segmentations for subword regularizationsubword-sampling
for _ in range(3):
print(sp.encode('New York', out_type=str,
enable_sampling=True, alpha=0.1, nbest_size=-1))
# different segmentations of the same text each callSampling is for training-time augmentation only; keep deterministic encoding at inference or your ids will not match run to run.
Inspect the vocabularyvocab-inspection
sp.get_piece_size() # vocab size
sp.id_to_piece(209) # piece string for an id
sp.piece_to_id('\u2581the') # id for a piece
sp.is_unknown(sp.piece_to_id('zzzzq'))Unknown text maps to the unk id (0 by default) unless you trained with byte_fallback=True, which decomposes unseen characters into byte pieces instead.
Train from an iterator without files on disktrain-from-iterator
import io
import sentencepiece as spm
model = io.BytesIO()
spm.SentencePieceTrainer.train(
sentence_iterator=iter(sentences),
model_writer=model,
vocab_size=8000,
)
sp = spm.SentencePieceProcessor(model_proto=model.getvalue())sentence_iterator plus model_writer replaces input/model_prefix; handy for streaming from a dataset library without dumping text files.
Ship the model as bytes instead of a file pathload-model-from-bytes
with open('m.model', 'rb') as f:
blob = f.read()
sp = spm.SentencePieceProcessor(model_proto=blob)Everything (normalization rules, vocab, merges) is inside the one protobuf blob, so the same bytes give identical tokenization in C++, Python or Go.
Train on CJK or mixed-language corporamultilingual-training
spm.SentencePieceTrainer.train(
input='multilingual.txt',
model_prefix='multi',
vocab_size=32000,
character_coverage=0.9995,
byte_fallback=True,
input_sentence_size=2000000,
shuffle_input_sentence=True,
)character_coverage=0.9995 is the usual setting for large character sets like Japanese; 1.0 for small alphabets. byte_fallback avoids unk on rare characters.
Reserve special tokens at training timeuser-defined-symbols
spm.SentencePieceTrainer.train(
input='corpus.txt',
model_prefix='m',
vocab_size=8000,
user_defined_symbols=['<sep>', '<cls>', '<mask>'],
)
sp = spm.SentencePieceProcessor(model_file='m.model')
sp.piece_to_id('<sep>') # fixed low id, never splituser_defined_symbols are extracted as whole tokens wherever they appear in text; control_symbols are reserved ids that never match raw text at all.
Encode a batch across threadsbatch-encode-threads
texts = load_million_lines()
ids = sp.encode(texts, out_type=int, num_threads=8)
# returns list[list[int]] in input orderPass the whole list in one call; the C++ side parallelizes, but converting results back to Python objects is GIL-bound, so scaling flattens at high thread counts.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| tokenizers | PyPI | Rust tokenizers from Hugging Face when you want BPE/WordPiece/Unigram training plus first-class transformers integration |
| tiktoken | PyPI | Fast byte-level BPE that matches OpenAI model vocabularies for encode-heavy workloads |
| subword-nmt | PyPI | The original minimal BPE scripts when you want something small and hackable for MT research |