PyPIAI / MLupdated 05 Aug 2026

tiktoken

Item: tiktoken
Rating: 3.5
Author: Keyoor

OpenAI's open source BPE tokenizer, written in Rust with Python bindings. It converts text into the exact token ids OpenAI models use, which is what you need to count tokens before hitting a context limit, estimate API costs, or truncate prompts precisely. You grab an encoding by name (o200k_base, cl100k_base) or via encoding_for_model("gpt-4o"), then encode and decode losslessly. OpenAI's own benchmark measures it 3-6x faster than a comparable open source tokenizer.

tiktoken · v0.13.0 · 54.4M/wk · ★ 18,915 · pushed 24 May 2026 · openai/tiktoken · docs · reviewed by Keyoor

Verdict

For anything touching OpenAI models it is the standard, and there is no reason to use anything else. It is deliberately narrow though: treat it as an OpenAI-only tool, not a general tokenization library.

API stability5/5The public surface is a couple of functions plus the Encoding class and it has barely changed since 2023; releases mostly add new encodings rather than alter behavior.

Docs2/5A short README, a pointer to docstrings in tiktoken/core.py, and a Cookbook notebook; fine for the happy path, but anything past encode and decode means reading source.

Maintenance3/5Last push about ten weeks before this review, and releases track OpenAI's model launches rather than a steady cadence; issues sit, but the code is small and rarely breaks.

Ecosystem4/5The default token counter across LLM tooling; frameworks, RAG chunkers, and cost calculators all depend on it, and 54M weekly downloads show it, though its reach ends at OpenAI encodings.

Use it if

You call the OpenAI API and need accurate token counts for cost estimates, context-window budgeting, or chunking text for embeddings
You need to truncate or split text at exact token boundaries instead of guessing with character counts
You tokenize large corpora and speed matters; the Rust core with threaded batch encoding is where the 3-6x benchmark shows up
You build rate limiting or billing logic around OpenAI usage and cannot afford approximation drift

Skip it if

Your model is not from OpenAI: tiktoken ships only OpenAI's encodings, so counts for Llama, Mistral, Gemma, or Claude are simply wrong; use the model's own tokenizer, or Anthropic's count-tokens API for Claude
You expect documentation: the README literally points you at tiktoken/core.py and an OpenAI Cookbook notebook, and there is no docs site
You need day-one support for new models; the model-to-encoding map lives inside the package, so encoding_for_model raises KeyError for models newer than your installed version, and releases are infrequent
You deploy air-gapped: encoding files are downloaded from OpenAI's CDN on first use and cached, which fails offline unless you pre-seed the cache

Setup reality

pip install tiktoken is painless on mainstream platforms thanks to prebuilt wheels; unusual platforms need a Rust toolchain to build from source. The surprise comes at runtime: the first get_encoding call downloads the encoding file from OpenAI's CDN and caches it (set TIKTOKEN_CACHE_DIR to control where), so locked-down or offline environments fail until you vendor the files. Also budget for the model-name mapping lag: after a new OpenAI model ships you may need to upgrade tiktoken before encoding_for_model recognizes it.

Patterns

Encode and decode textbasic-encode-decode

import tiktoken

enc = tiktoken.get_encoding("o200k_base")
tokens = enc.encode("hello world")
assert enc.decode(tokens) == "hello world"

o200k_base is the GPT-4o era encoding; cl100k_base covers GPT-4 and GPT-3.5. Counts differ between them, so pick by model, not by habit.

Get the encoding for an API modelencoding-for-model

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")
print(enc.name)  # o200k_base

Raises KeyError for model names newer than your installed tiktoken; catch it and fall back to a sensible default encoding instead of crashing.

Count tokens to estimate costcount-tokens-cost

enc = tiktoken.encoding_for_model("gpt-4o")
n = len(enc.encode(prompt))
usd = n / 1_000_000 * price_per_mtok

For chat completions the billed count also includes per-message overhead tokens the API adds around your content, so treat client-side counts as close, not exact.

Truncate text to a token budgettruncate-to-budget

def truncate(text: str, enc, max_tokens: int) -> str:
    tokens = enc.encode(text)
    if len(tokens) <= max_tokens:
        return text
    return enc.decode(tokens[:max_tokens])

Decoding a sliced token list can end mid-word or lose bytes on multibyte characters; decode is only guaranteed lossless on complete valid sequences.

Encode a corpus in parallelbatch-encode-threads

enc = tiktoken.get_encoding("o200k_base")
all_tokens = enc.encode_batch(docs, num_threads=8)

encode_batch runs in the Rust core across threads, so this is real parallelism, not GIL-bound; looping encode in Python is the slow way.

Handle special tokens in inputspecial-tokens

enc = tiktoken.get_encoding("cl100k_base")

enc.encode("hi <|endoftext|>")  # raises ValueError
enc.encode_ordinary("hi <|endoftext|>")  # treats it as plain text
enc.encode("<|endoftext|>", allowed_special={"<|endoftext|>"})

encode raises on special-token text by default because untrusted input containing them is a prompt-injection vector; encode_ordinary is usually what you want for user text.

See how text splits into tokensinspect-tokens

enc = tiktoken.get_encoding("o200k_base")
for t in enc.encode("Tokenization"):
    print(t, enc.decode_single_token_bytes(t))

decode_single_token_bytes returns bytes, not str, because a single token can be a partial UTF-8 sequence.

Pre-seed the cache for offline deploysoffline-cache

# on a machine with internet:
TIKTOKEN_CACHE_DIR=./tiktoken-cache \
  python -c "import tiktoken; tiktoken.get_encoding('o200k_base')"

# ship the folder with your image, then:
export TIKTOKEN_CACHE_DIR=/app/tiktoken-cache

Without a warmed cache the first get_encoding call fetches from OpenAI's CDN at runtime; in egress-restricted deploys that is a production crash, not a laptop problem.

Extend an encoding with your tokenscustom-encoding

import tiktoken

base = tiktoken.get_encoding("cl100k_base")
enc = tiktoken.Encoding(
    name="cl100k_im",
    pat_str=base._pat_str,
    mergeable_ranks=base._mergeable_ranks,
    special_tokens={
        **base._special_tokens,
        "<|im_start|>": 100264,
        "<|im_end|>": 100265,
    },
)

This is straight from the README, which itself flags the underscore attributes as private; pin your tiktoken version if you rely on them. The tiktoken_ext plugin mechanism is the route if get_encoding must find your encoding by name.

Visualize how BPE workseducational-bpe

from tiktoken._educational import SimpleBytePairEncoding

enc = SimpleBytePairEncoding.from_tiktoken("cl100k_base")
enc.encode("hello world aaaaaaaaaaaa")

The _educational module prints merge steps as they happen; great for teaching BPE, far too slow for real workloads.

Alternatives

Package	Registry	Pick it when
tokenizers	PyPI	You tokenize for Hugging Face models; it loads each model's actual tokenizer instead of OpenAI's encodings
transformers	PyPI	You want AutoTokenizer convenience for any Hub model, so counts match the model you actually call
anthropic	PyPI	You need Claude token counts; there is no public local Claude tokenizer, so the SDK's count-tokens endpoint is the supported way