nltk
NLTK (Natural Language Toolkit) is the classic Python suite for natural language processing: tokenizers, stemmers, lemmatizers, part-of-speech taggers, parsers, a named-entity chunker, the VADER sentiment analyzer, and access to dozens of corpora and lexical resources like WordNet. It dates to 2001 and was built for teaching and research, with a free companion O'Reilly book. Nothing in it is neural; it is rule-based and classical statistical NLP. That makes it light, transparent, and great for learning how NLP works, and also why production teams have largely moved to spaCy or transformer models for accuracy-critical work.
Still the best way to learn NLP and a fine toolbox for classic text-processing chores, backed by 25 years of steady maintenance. For production accuracy on tagging, NER, or anything multilingual, reach for spaCy or stanza instead.
Use it if
- You are learning NLP or teaching it; the toolkit, the corpora, and the free book at nltk.org/book form the best guided introduction that exists
- You need classic text-processing primitives: tokenization, stopword lists, stemming, n-grams, frequency distributions, or WordNet lookups without loading a neural model
- You need quick lexicon-based sentiment (VADER) on social-media-style text where a rules approach is good enough and speed matters
- You work with the linguistic corpora NLTK ships access to (Brown, Gutenberg, treebanks, WordNet) for research or coursework
- You need production-grade accuracy for NER, tagging, or parsing; spaCy and transformer models beat NLTK's classical models by a wide margin and run faster on pipelines
- You hate runtime data downloads: almost nothing works until nltk.download() fetches the right datasets, and the punkt to punkt_tab rename plus the _eng tagger suffix broke a decade of Stack Overflow answers
- Your text is not English; NLTK's non-English coverage is thin and uneven compared to spaCy or stanza
- You are building anything LLM-adjacent; modern tokenization for LLM work happens in libraries like tiktoken or Hugging Face tokenizers, not NLTK
Setup reality
pip install nltk is trivial (needs Python 3.10+), but the package is a shell until you download data. Every tutorial's first error is LookupError: Resource punkt_tab not found, because tokenizers, taggers, stopwords, and WordNet are all separate downloads. Since 3.8.2 the sentence tokenizer data is punkt_tab (the old pickle-based punkt was dropped for security reasons), and since 3.9 the POS tagger resource is averaged_perceptron_tagger_eng with the language suffix. Older answers cite the old names and fail. On servers you must pre-download data into a path NLTK searches, or set NLTK_DATA.
Patterns
Download the data most tasks needdownload-required-data
import nltk
nltk.download("punkt_tab")
nltk.download("stopwords")
nltk.download("wordnet")
nltk.download("averaged_perceptron_tagger_eng")Nothing works without this step. Note the current names: punkt_tab (not punkt) since 3.8.2 and averaged_perceptron_tagger_eng (with _eng) since 3.9. Old tutorials use the outdated names.
Tokenize text into wordsword-tokenize
from nltk.tokenize import word_tokenize
text = "NLTK isn't dead, it's just classical."
tokens = word_tokenize(text)
# ['NLTK', 'is', "n't", 'dead', ',', 'it', "'s", 'just', 'classical', '.']Requires punkt_tab. Contractions split into two tokens (is + n't), which surprises people counting words.
Split text into sentencessentence-tokenize
from nltk.tokenize import sent_tokenize
text = "Dr. Smith arrived. He was late."
print(sent_tokenize(text))
# ['Dr. Smith arrived.', 'He was late.']The Punkt algorithm handles abbreviations like Dr. reasonably well for English; pass language="german" etc. for other supported languages.
Filter out stopwordsremove-stopwords
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
stops = set(stopwords.words("english"))
tokens = word_tokenize("This is a simple example of filtering")
kept = [t for t in tokens if t.lower() not in stops]
# ['simple', 'example', 'filtering']Build the set once, not inside a loop; stopwords.words() re-reads the corpus file each call and is slow if repeated.
Stem words with PorterStemmerstem-words
from nltk.stem import PorterStemmer
stemmer = PorterStemmer()
print([stemmer.stem(w) for w in ["running", "flies", "studies"]])
# ['run', 'fli', 'studi']Stems are not words ('fli', 'studi'). Use stemming for matching and search, lemmatization when you need real dictionary forms.
Lemmatize with WordNetlemmatize-words
from nltk.stem import WordNetLemmatizer
lem = WordNetLemmatizer()
print(lem.lemmatize("running")) # running (default pos is noun)
print(lem.lemmatize("running", pos="v")) # runThe default part of speech is noun, so verbs pass through unchanged unless you pass pos="v". This is the number one lemmatizer complaint.
Part-of-speech tag a sentencepos-tagging
from nltk import pos_tag
from nltk.tokenize import word_tokenize
tokens = word_tokenize("The quick brown fox jumps")
print(pos_tag(tokens))
# [('The', 'DT'), ('quick', 'JJ'), ('brown', 'JJ'), ('fox', 'NN'), ('jumps', 'VBZ')]Needs averaged_perceptron_tagger_eng downloaded. Tags are Penn Treebank codes; nltk.help.upenn_tagset('NN') explains any of them.
Extract named entitiesnamed-entity-recognition
import nltk
nltk.download("maxent_ne_chunker_tab")
nltk.download("words")
from nltk import pos_tag, ne_chunk
from nltk.tokenize import word_tokenize
tree = ne_chunk(pos_tag(word_tokenize("Ada Lovelace worked in London")))
print(tree)The chunker data is maxent_ne_chunker_tab in current releases. Accuracy is mediocre by modern standards; use spaCy if NER quality actually matters.
Count word frequenciesfrequency-distribution
from nltk import FreqDist
from nltk.tokenize import word_tokenize
tokens = word_tokenize("to be or not to be")
fd = FreqDist(t.lower() for t in tokens)
print(fd.most_common(2)) # [('to', 2), ('be', 2)]FreqDist is a Counter subclass with NLP extras like fd.plot() and fd.hapaxes() for words that occur once.
Generate bigrams and trigramsngrams
from nltk import ngrams
from nltk.tokenize import word_tokenize
tokens = word_tokenize("natural language processing with python")
print(list(ngrams(tokens, 2)))
# [('natural', 'language'), ('language', 'processing'), ...]ngrams returns a generator, so wrap it in list() if you need to iterate twice or inspect it.
Lexicon-based sentiment with VADERvader-sentiment
import nltk
nltk.download("vader_lexicon")
from nltk.sentiment import SentimentIntensityAnalyzer
sia = SentimentIntensityAnalyzer()
print(sia.polarity_scores("This library is great but the setup is annoying"))
# {'neg': ..., 'neu': ..., 'pos': ..., 'compound': ...}VADER was tuned for social media text and handles negation, caps, and emoji. Use the compound score with cutoffs around +/-0.05 for a three-way label.
Look up synonyms in WordNetwordnet-synonyms
from nltk.corpus import wordnet
syns = wordnet.synsets("car")
print(syns[0].definition())
print({l.name() for s in syns for l in s.lemmas()})Requires the wordnet download. Synsets are sense-level, so 'car' includes cable car and railway car senses; filter by definition when precision matters.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| spacy | PyPI | You want fast, accurate, production-oriented pipelines for tagging, NER, and parsing |
| stanza | PyPI | You need high-accuracy neural models across 60+ languages from the Stanford NLP group |
| textblob | PyPI | You want an even simpler API for quick sentiment and noun-phrase extraction on top of NLTK-style tooling |