spacy
spaCy is Explosion's NLP library for Python, written largely in Cython for speed. You load a pretrained pipeline (tokenizer, tagger, parser, named entity recognizer, lemmatizer) and feed it text; it returns a Doc object where every token carries part of speech, dependency relation, lemma, and entity labels. Pipelines exist for 70+ languages, rule-based matchers combine with the statistical models, and there is a full training system with config files if you need custom components. It is built for shipping products, not for research notebooks.
Still the default for fast, structured NLP on CPU and the API is a pleasure, but the project is coasting: check first whether an LLM call or a fine-tuned transformer makes your pipeline unnecessary before you invest in one.
Use it if
- You need structured linguistic output (entities, part-of-speech tags, dependency parses, lemmas) from large volumes of text at CPU speed
- You are processing millions of documents and want nlp.pipe batching with multiprocessing instead of per-document Python overhead
- You want to mix hand-written rules (Matcher, PhraseMatcher, EntityRuler) with trained models in one pipeline
- You need consistent tokenization and sentence segmentation across many languages behind a single API
- You need best-possible accuracy: fine-tuned models from the transformers library beat spaCy's small CPU pipelines on most benchmarks, and bolting spacy-transformers on drags in torch anyway
- Development has slowed noticeably: the 3.x line has been current since 2021, releases are mostly maintenance fixes, and the long-promised v4 still has not shipped, so do not count on major new features
- Your task is classification or extraction over a modest number of documents: a single LLM API call per document is often simpler and more accurate than training and maintaining a spaCy pipeline
- You only need tokenization for an LLM workflow: tiktoken or the tokenizers library is lighter and matches what the model actually sees
Setup reality
pip install spacy gets you prebuilt wheels on mainstream platforms, but the library is useless without a model, and models are separate downloads: python -m spacy download en_core_web_sm pip-installs a package from a GitHub release URL. That is awkward behind corporate proxies and breaks naive lockfiles, so pin models with the direct wheel URL in requirements. The compiled dependency stack (thinc, cymem, blis, srsly) means brand-new Python releases sometimes wait weeks for wheels. GPU use requires the right spacy[cuda...] extra and a matching CuPy, and transformer pipelines pull the full torch stack.
Patterns
Load a pipeline and extract named entitiesload-pipeline-extract-entities
import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("Apple is looking at buying U.K. startup for $1 billion")
for ent in doc.ents:
print(ent.text, ent.label_, ent.start_char, ent.end_char)The model must be downloaded first: python -m spacy download en_core_web_sm. spacy.load raises OSError if it is missing.
Read per-token linguistic attributestoken-attributes
import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("She was reading the paper quickly.")
for token in doc:
print(token.text, token.lemma_, token.pos_, token.dep_, token.is_stop)Underscore attributes (lemma_, pos_) are the human-readable strings; the versions without underscore are integer hash IDs.
Process many texts efficiently with nlp.pipebatch-processing
import spacy
nlp = spacy.load("en_core_web_sm")
texts = ["First document.", "Second document.", "Third document."]
for doc in nlp.pipe(texts, batch_size=200, n_process=2):
print([ent.text for ent in doc.ents])nlp.pipe is far faster than calling nlp() in a loop. n_process forks workers, which costs model-load time per worker; only worth it for large jobs.
Disable pipeline components you do not needdisable-components
import spacy
nlp = spacy.load("en_core_web_sm", disable=["parser", "lemmatizer"])
doc = nlp("Just need entities here.")
print(doc.ents)
# or temporarily:
with nlp.select_pipes(enable=["tok2vec", "ner"]):
doc = nlp("Entities only, parser skipped.")The parser is the slowest component. Disabling unused components is the single biggest speedup available.
Split text into sentencessentence-segmentation
import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("This is one sentence. Here is another one!")
for sent in doc.sents:
print(sent.text)doc.sents needs the parser or the senter component. For speed-only segmentation, use the lightweight senter and disable the parser.
Match token patterns with the Matcherrule-based-matcher
import spacy
from spacy.matcher import Matcher
nlp = spacy.load("en_core_web_sm")
matcher = Matcher(nlp.vocab)
pattern = [{"LOWER": "machine"}, {"LOWER": "learning"}]
matcher.add("ML", [pattern])
doc = nlp("Machine learning beats manual rules, except when it doesn't.")
for match_id, start, end in matcher(doc):
print(doc[start:end].text)Patterns operate on token attributes, not raw regex over the string. Each dict matches exactly one token.
Match large terminology lists with PhraseMatcherphrase-matcher
import spacy
from spacy.matcher import PhraseMatcher
nlp = spacy.load("en_core_web_sm")
matcher = PhraseMatcher(nlp.vocab, attr="LOWER")
terms = ["acetaminophen", "ibuprofen", "naproxen sodium"]
matcher.add("DRUG", [nlp.make_doc(t) for t in terms])
doc = nlp("She took Ibuprofen and naproxen sodium.")
for match_id, start, end in matcher(doc):
print(doc[start:end].text)Use nlp.make_doc for the patterns to skip the full pipeline; PhraseMatcher scales to hundreds of thousands of terms.
Add custom entities with the EntityRulercustom-entity-ruler
import spacy
nlp = spacy.load("en_core_web_sm")
ruler = nlp.add_pipe("entity_ruler", before="ner")
ruler.add_patterns([
{"label": "PRODUCT", "pattern": "MacBook Pro"},
{"label": "ORG", "pattern": [{"LOWER": "acme"}, {"LOWER": "corp"}]},
])
doc = nlp("Acme Corp shipped a MacBook Pro.")
print([(e.text, e.label_) for e in doc.ents])Placement matters: before="ner" lets your rules win over the statistical NER when spans overlap.
Compare texts with word vectorsword-vectors-similarity
import spacy
nlp = spacy.load("en_core_web_md") # md/lg models include vectors
doc1 = nlp("I like salty fries and hamburgers.")
doc2 = nlp("Fast food tastes very good.")
print(doc1.similarity(doc2))The sm models have no real word vectors and similarity on them is nearly meaningless. For serious semantic similarity use sentence-transformers instead.
Register a custom pipeline componentcustom-pipeline-component
import spacy
from spacy.language import Language
@Language.component("stats_logger")
def stats_logger(doc):
print(f"{len(doc)} tokens, {len(doc.ents)} entities")
return doc
nlp = spacy.load("en_core_web_sm")
nlp.add_pipe("stats_logger", last=True)
nlp("Google was founded in 1998 in California.")Components must accept and return the Doc. Register with a string name so the pipeline stays serializable.
Save and reload processed Docs with DocBinserialize-docs
import spacy
from spacy.tokens import DocBin
nlp = spacy.load("en_core_web_sm")
doc_bin = DocBin(store_user_data=True)
for doc in nlp.pipe(["First text.", "Second text."]):
doc_bin.add(doc)
doc_bin.to_disk("corpus.spacy")
# later
docs = list(DocBin().from_disk("corpus.spacy").get_docs(nlp.vocab))DocBin is the format spaCy's training system expects, and far cheaper than pickling Doc objects individually.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| nltk | PyPI | You are teaching or prototyping classic NLP algorithms and want corpora and reference implementations over production speed. |
| stanza | PyPI | You want Stanford's neural pipelines and will trade speed for higher accuracy on parsing and tagging in many languages. |
| transformers | PyPI | You need state-of-the-art accuracy from fine-tuned models and have the GPU budget to run them. |