tokenizers
Hugging Face's tokenization engine: a Rust core with Python bindings implementing the tokenizer pipelines behind most modern models (BPE, WordPiece, Unigram). It both runs pretrained tokenizers (every tokenizer.json on the Hub) and trains new ones from raw text, tokenizing about a GB of text in under 20 seconds on a server CPU. It tracks alignments through normalization, so every token maps back to exact character offsets in the original string, and it handles truncation, padding, and special-token insertion as part of the pipeline.
The engine nearly everyone already uses through transformers. Reach for it directly when you train custom tokenizers, need offsets, or want Rust speed in a pipeline; otherwise let AutoTokenizer drive it for you.
Use it if
- You train a custom tokenizer on your own corpus (domain text, code, a new language); the trainers make it a few lines and it is fast enough to iterate on vocab size
- You need character offsets: NER, span extraction, and highlighting require mapping tokens back to the original text, and offset tracking through normalization is a core feature here
- You tokenize at scale outside model code (data pipelines, preprocessing jobs) and want Rust throughput without importing all of transformers
- You ship tokenization in another runtime; the same tokenizer.json loads from the Rust, Python, and Node bindings
- You just want to tokenize for an existing model inside normal model code: transformers' AutoTokenizer wraps this library and adds the model-specific conventions (padding side, chat templates, slow-tokenizer fallback) you would otherwise reimplement
- You only count OpenAI tokens; tiktoken ships OpenAI's actual encodings and is the right tool there
- You expect polished docs: the quicktour is fine, but component-level behavior (normalizers, pre-tokenizers, post-processors) often means reading Rust source or GitHub issues, and errors bubble up from Rust with messages that stop at the binding boundary
Setup reality
pip install tokenizers works from prebuilt wheels on mainstream platforms; only unusual ones need a Rust toolchain, with source installs going through the bindings/python subdirectory of the repo. The classic runtime gotcha is the fork warning: use Rust parallelism and then fork (as PyTorch DataLoader workers do) and it prints 'The current process just got forked' and disables parallelism until you set TOKENIZERS_PARALLELISM=false or restructure. Version drift bites too: a tokenizer.json saved by a newer release can refuse to load in an older one, so pin the same version across training and serving.
Patterns
Load a tokenizer from the Hubload-pretrained
from tokenizers import Tokenizer
tok = Tokenizer.from_pretrained("bert-base-uncased")
out = tok.encode("Hello, world!")
print(out.tokens, out.ids)from_pretrained needs a tokenizer.json in the repo; models that only ship slow-tokenizer files fail here even though AutoTokenizer would fall back for you.
Train a BPE tokenizer from filestrain-bpe
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace
tok = Tokenizer(BPE(unk_token="[UNK]"))
tok.pre_tokenizer = Whitespace()
trainer = BpeTrainer(
vocab_size=30000,
special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"],
)
tok.train(files=["corpus.txt"], trainer=trainer)Forgetting the pre_tokenizer is the classic mistake: BPE then merges across whitespace and produces a vocabulary full of multi-word junk tokens.
Save and reload as one JSON filesave-load
tok.save("tokenizer.json")
from tokenizers import Tokenizer
tok = Tokenizer.from_file("tokenizer.json")One file holds the whole pipeline from normalizer to post-processor. Load it with the same or newer library version; older versions can reject files saved by newer ones.
Encode a batch in parallelbatch-encode
outs = tok.encode_batch(["first text", "second text"])
lens = [len(o.ids) for o in outs]encode_batch parallelizes across the batch in Rust; looping encode from Python throws that advantage away.
Pad and truncate to fixed lengthpadding-truncation
tok.enable_padding(pad_id=0, pad_token="[PAD]", length=128)
tok.enable_truncation(max_length=128)
outs = tok.encode_batch(texts)These settings are stateful on the tokenizer object and get baked into save(); call no_padding() and no_truncation() when you need raw lengths back.
Map tokens back to source textoffsets-alignment
text = "Hugging Face is based in NYC"
out = tok.encode(text)
for token, (start, end) in zip(out.tokens, out.offsets):
print(token, repr(text[start:end]))Offsets survive normalization steps like lowercasing and accent stripping, which is exactly what regex-based alignment hacks get wrong.
Add special tokens via post-processortemplate-processing
from tokenizers.processors import TemplateProcessing
tok.post_processor = TemplateProcessing(
single="[CLS] $A [SEP]",
pair="[CLS] $A [SEP] $B [SEP]",
special_tokens=[("[CLS]", 1), ("[SEP]", 2)],
)The ids must match your vocab (check tok.token_to_id); nothing validates them at assignment, so a mismatch shows up later as silently wrong ids.
Decode ids back to textdecode
ids = tok.encode("Hello, world!").ids
print(tok.decode(ids))
print(tok.decode(ids, skip_special_tokens=False))decode drops special tokens by default; keep them when debugging what the model actually saw.
Silence the fork warning correctlyfork-parallelism
import os
os.environ["TOKENIZERS_PARALLELISM"] = "false"
# set before forking, e.g. before a DataLoader with num_workers > 0The 'current process just got forked' warning means Rust threads were spawned pre-fork; parallelism silently turns off in children, so either set this or tokenize before forking on purpose.
Extend the vocabularyadd-tokens
tok.add_special_tokens(["<|user|>", "<|assistant|>"])
tok.add_tokens(["domain_term"])
print(tok.get_vocab_size())Adding tokens changes the vocab size; the matching model's embedding matrix must be resized too or you get index errors at train time.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| transformers | PyPI | You want a model's tokenizer with all its conventions handled; AutoTokenizer drives this library underneath |
| tiktoken | PyPI | You count or produce tokens for OpenAI models specifically |
| sentencepiece | PyPI | You need Google's original Unigram/BPE trainer that some model families (T5, the Llama lineage) were built on |