lxml
lxml wraps the C libraries libxml2 and libxslt behind the familiar ElementTree API, making it the fast, full-featured way to parse, query, and build XML and HTML in Python. You get real XPath 1.0, XSLT transforms, XML Schema and DTD validation, a forgiving HTML parser, and streaming iterparse for files too big for memory. It is what BeautifulSoup, pandas.read_html, and scrapy's selectors run on when speed matters, and it has been the default answer for serious XML work in Python for two decades.
Still the definitive XML/HTML workhorse for Python: fastest mainstream option, only one with real XPath and XSLT, and stable for twenty years. Use stdlib ElementTree for trivial jobs, add defusedxml habits for untrusted input, and know that you are depending on a nearly one-person project.
Use it if
- You need XPath: stdlib ElementTree supports only a tiny subset, while lxml gives you full XPath 1.0 with functions, predicates, and namespace handling
- You process large feeds or exports (multi-GB XML): iterparse plus element clearing keeps memory flat where DOM parsers fall over
- You scrape or clean real-world HTML at volume: the HTML parser tolerates broken markup and is several times faster than pure-Python parsers
- You need XSLT, XML Schema, or DTD validation, which practically no other maintained Python library does natively
- Your XML is small and well-formed: xml.etree.ElementTree ships with Python, has the same core API, and spares you a compiled dependency
- You parse untrusted XML without reading the security docs: entity expansion and external entity (XXE) attacks are real; you must configure the parser defensively, and defusedxml exists precisely because people skip this
- You mostly extract text and attributes from HTML: selectolax is faster and lighter for that one job, and BeautifulSoup is friendlier for messy exploratory scraping
- You deploy where wheels do not exist (Alpine/musl variants, niche architectures): building from source needs libxml2 and libxslt dev headers plus a compiler, a classic CI time sink
- The bus factor worries you: it is essentially maintained by one person funded by small donations, documented income of a few thousand euros per year against 250M+ monthly downloads
Setup reality
On CPython with common platforms, pip install lxml grabs a binary wheel and just works. Off that path (Alpine, FreeBSD, PyPy without wheels) you need libxml2-dev, libxslt-dev, and a C toolchain, and version mismatches between system libxml2 and what lxml expects produce confusing crashes. Note lxml 6.x removed support for Pythons older than 3.8, and while the metadata still allows 3.8, wheels concentrate on current CPython releases. API-wise the recurring stumbles are namespaces (you must use {uri}tag or namespace maps everywhere) and the fact that Element.text/tail splitting of mixed content confuses everyone at first.
Patterns
Parse XML and query with XPathparse-and-xpath
from lxml import etree
tree = etree.parse("catalog.xml")
for title in tree.xpath("//book[@lang='en']/title/text()"):
print(title)xpath() returns a list of strings, elements, or numbers depending on the expression; this full XPath support is the main upgrade over stdlib ElementTree.
Parse messy HTML from a stringparse-html-string
from lxml import html
doc = html.fromstring(page_bytes)
links = [(a.text_content(), a.get("href")) for a in doc.xpath("//a[@href]")]The HTML parser repairs broken markup instead of raising; feed it raw bytes so encoding is detected from meta tags rather than guessed.
Query namespaced XMLxpath-with-namespaces
from lxml import etree
tree = etree.parse("feed.xml")
ns = {"a": "http://www.w3.org/2005/Atom"}
for entry in tree.xpath("//a:entry/a:title/text()", namespaces=ns):
print(entry)There is no null-prefix default in XPath: elements in a default namespace still need a prefix in your expression, the number one lxml stumbling block.
Stream a multi-GB file with iterparsestream-huge-files
from lxml import etree
for _, elem in etree.iterparse("huge.xml", tag="{urn:example}record"):
process(elem)
elem.clear()
while elem.getprevious() is not None:
del elem.getparent()[0]Without clear() plus deleting processed siblings, iterparse still builds the whole tree and memory climbs to DOM levels; the cleanup dance is mandatory.
Build and serialize a documentbuild-document
from lxml import etree
root = etree.Element("order", id="42")
item = etree.SubElement(root, "item", sku="A-1")
item.text = "Widget"
xml_bytes = etree.tostring(root, xml_declaration=True, encoding="UTF-8", pretty_print=True)tostring returns bytes when an encoding is given and str only for encoding='unicode'; mixing those up is a classic TypeError source.
Parse untrusted XML defensivelysecure-untrusted-parse
from lxml import etree
parser = etree.XMLParser(
resolve_entities=False,
no_network=True,
dtd_validation=False,
load_dtd=False,
huge_tree=False,
)
tree = etree.fromstring(untrusted_bytes, parser)This blocks external entity (XXE) resolution and network fetches; if security review matters, defusedxml documents the full threat list for XML parsers.
Use CSS selectors instead of XPathcss-selectors
from lxml import html
doc = html.fromstring(page)
for card in doc.cssselect("div.product > h2 a"):
print(card.get("href"))cssselect requires the separate cssselect package (pip install cssselect); it compiles selectors to XPath under the hood, so complex pseudo-selectors are unsupported.
Validate against an XML Schemavalidate-schema
from lxml import etree
schema = etree.XMLSchema(etree.parse("order.xsd"))
doc = etree.parse("order.xml")
if not schema.validate(doc):
for err in schema.error_log:
print(err.line, err.message)validate() returns a bool and fills error_log; use schema.assertValid(doc) when you would rather get a raised exception with the first error.
Apply an XSLT transformxslt-transform
from lxml import etree
transform = etree.XSLT(etree.parse("to_html.xsl"))
result = transform(etree.parse("data.xml"))
print(str(result))This is XSLT 1.0 (libxslt); stylesheets written for XSLT 2.0/3.0 processors like Saxon will not run.
Find, modify, and remove elements in placemodify-tree
from lxml import etree
tree = etree.parse("config.xml")
root = tree.getroot()
for node in root.findall(".//debug"):
node.getparent().remove(node)
root.find(".//timeout").text = "30"
tree.write("config.xml", encoding="UTF-8", xml_declaration=True)Unlike stdlib ElementTree, every lxml element knows getparent(), which is what makes in-place removal a one-liner.
Extract readable text from HTMLextract-clean-text
from lxml import html
doc = html.fromstring(page)
for bad in doc.xpath("//script | //style"):
bad.getparent().remove(bad)
text = doc.text_content()text_content() concatenates all descendant text including script bodies, so strip script and style nodes first or your 'article text' includes JavaScript.
Use lxml as BeautifulSoup's enginesoup-fallback-parse
from bs4 import BeautifulSoup
soup = BeautifulSoup(page_bytes, "lxml")
print(soup.select_one("h1").get_text(strip=True))Passing 'lxml' as the parser gives BeautifulSoup roughly an order-of-magnitude parse speedup over html.parser while keeping its friendlier API.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| beautifulsoup4 | PyPI | You want forgiving, ergonomic HTML exploration and are fine being slower; it can use lxml as its parser underneath. |
| selectolax | PyPI | You only need fast CSS-selector extraction from HTML and want a smaller, quicker dependency than a full XML stack. |
| defusedxml | PyPI | You parse untrusted XML and want safe-by-default wrappers instead of hand-configuring parser hardening. |
| xmltodict | PyPI | You just want small XML converted to Python dicts and back without learning a tree API. |