html5lib review
html5lib 1.1 is a pure-Python implementation of the WHATWG HTML parsing algorithm. Feed it broken markup and it applies browser-style recovery rules for implied document nodes, misnested formatting, table insertion modes, encodings, and context-sensitive fragments. Output defaults to `xml.etree`, with minidom and lxml builders available, and a companion serializer turns supported trees back into HTML. The current 1.1 release dates to 2020; it removed the optional datrie accelerator, stopped supporting Python 3.3 and 3.4, and deprecated its sanitizer. Our Python 3.12 import worked, but the package has no `py.typed` marker.
html5lib 1.1 installed in 0.2 seconds and used 1 MB across 3 packages in our sandbox, but the current release is from June 2020 and has no `py.typed` marker. Keep it for browser-like repair of difficult HTML; do not choose it as a sanitizer or as the default engine for a throughput-bound crawler.
We installed it
| Install | ✓ · 0.2s | 3 packages on disk · 1 MB |
| Import | ✓ | import html5lib in 0.52s · pure Python · requires Python >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.* |
| Known vulns | 0 | (pip-audit) |
Answers from our run
Does html5lib install cleanly?
Yes. In a fresh container with an empty cache, pip install html5lib finished in 0.2s, leaving 3 packages and 1 MB on disk. pip-audit reported no known vulnerabilities.
What does html5lib need to run?
Python >=2.7, !=3.0., !=3.1., !=3.2., !=3.3., !=3.4.*, and nothing compiled: it is pure Python. In our run import html5lib succeeded in 0.52s.
html5lib or beautifulsoup4: which should you use?
beautifulsoup4: Use it for a friendlier selection and traversal API, choosing html5lib or another parser behind it per workload. html5lib 1.1 installed in 0.2 seconds and used 1 MB across 3 packages in our sandbox, but the current release is from June 2020 and has no py.typed marker.
When should you not use html5lib?
Parser throughput limits a large crawler. html5lib executes the tree-building algorithm in Python, while lxml and selectolax put their hot path in native code.
Use it if
- Malformed pages must be repaired into the same broad tree shape that a standards-following browser parser would produce.
- A Beautiful Soup extraction gets the wrong nesting with another backend and the exact problem page behaves correctly under `html5lib`.
- Native compilation is unavailable and slower pure-Python parsing is acceptable for the workload.
- Generated HTML needs conformance error codes and source positions, or strict parsing should fail at the first error.
- Parser throughput limits a large crawler. html5lib executes the tree-building algorithm in Python, while lxml and selectolax put their hot path in native code.
- You need to make attacker-controlled HTML safe. Version 1.1 deprecates its sanitizer and parsing alone does not remove dangerous URLs, attributes, or elements.
- Dependency policy requires regular releases. PyPI's current 1.1 artifact was uploaded in June 2020, although the repository was pushed in April 2026.
- A package-supplied typing contract is mandatory. Our install found no `py.typed`, so strict consumers need stubs or must treat the library as untyped.
- Your input is ordinary well-formed HTML and the main task is CSS or XPath selection. html5lib builds trees, while selector ergonomics come from another wrapper or builder.
Setup reality
Our install of html5lib 1.1 completed in 0.2 seconds in a clean Python 3.12 container. Three packages occupied 1 MB afterward, and importing html5lib took 0.52 seconds. The PyPI metadata lists 8 direct requirements when optional extras and markers are counted. It is pure Python, has no py.typed marker, and declares Python 2.7 or 3.5 and newer. pip-audit reported 0 known vulnerabilities.
The default xml.etree output uses the XHTML namespace. A paragraph tag is {http://www.w3.org/1999/xhtml}p, so tree.find('p') silently misses it. Pass namespaceHTMLElements=False for plain tag names or use namespace-qualified queries. Fragment parsing has a second context trap: parseFragment() assumes a generic container unless you provide one. Table rows and option elements can be moved or dropped when parsed under the wrong parent.
Give html5lib bytes when you want its HTML encoding algorithm. For an HTTP body, pass the response charset as transport_encoding instead of decoding first. The chardet extra is a fallback only when earlier signals do not settle the encoding. Choosing the lxml builder installs lxml separately. The README warns that html5lib plus lxml is unsupported on PyPy because that combination is known to segfault there.
Serialization is not a byte-for-byte round trip. By default it can omit optional html, head, and body tags, even when the parser inserted them. Set tag omission, quoting, attribute order, whitespace filtering, and error policy explicitly for stable snapshots. One HTMLParser instance keeps parse errors in parser.errors; use a fresh instance per document when reports must remain separate. Strict mode rejects common pages such as those missing a doctype, so reserve it for controlled output.
Patterns
Parse bytes into a complete document parse-complete-document
import html5lib
with open('page.html', 'rb') as source:
document = html5lib.parse(source)
print(document.tag)
# {http://www.w3.org/1999/xhtml}htmlBinary input lets version 1.1 apply the HTML encoding algorithm. The resulting tree includes implied `html`, `head`, and `body` elements even when the source omitted them.
Build ElementTree nodes with plain tag names disable-html-namespace
import html5lib
tree = html5lib.parse(
'<ul><li>first<li>second',
namespaceHTMLElements=False,
)
items = [node.text for node in tree.iter('li')]The default tag name includes the XHTML namespace, which makes `iter('li')` return no matches. Set `namespaceHTMLElements=False` only when downstream code expects unqualified names.
Parse a row inside table context parse-contextual-fragment
import html5lib
row = html5lib.parseFragment(
'<tr><td>one<td>two',
container='table',
namespaceHTMLElements=False,
)HTML fragment repair depends on the parent element. Without `container='table'`, version 1.1 applies generic fragment rules and can rearrange or discard row cells.
Select the optional lxml tree builder build-lxml-tree
import html5lib
parser = html5lib.HTMLParser(
tree=html5lib.getTreeBuilder('lxml'),
)
document = parser.parse('<p>Hello')
paragraphs = document.xpath('//html:p', namespaces={
'html': 'http://www.w3.org/1999/xhtml',
})lxml is a separate installation, not one of the 3 packages in our base sandbox. The html5lib README says this builder is unsupported on PyPy because it can segfault.
Inspect error codes and positions collect-parse-errors
import html5lib
parser = html5lib.HTMLParser()
parser.parse('<p>x</p></br>')
for position, code, details in parser.errors:
print(position, code, details)Version 1.1 stores errors on the parser object. Create a new parser for each document, or clear and label the list before reusing one across inputs.
Raise on the first conformance error fail-on-html-error
import html5lib
parser = html5lib.HTMLParser(strict=True)
try:
parser.parse('<p>missing doctype')
except html5lib.html5parser.ParseError as error:
print(error)Strict mode treats a missing doctype as an error, so many ordinary web pages fail immediately. Use this 1.1 option to test HTML you control, not as a default crawler setting.
Preserve optional tags during serialization serialize-document-tags
import html5lib
document = html5lib.parse('<p>Hello <b>there')
html = html5lib.serialize(
document,
tree='etree',
omit_optional_tags=False,
)
print(html)The serializer normally removes optional document tags. `omit_optional_tags=False` keeps the repaired `html`, `head`, and `body` structure visible in output and snapshots.
Use explicit serializer rules for snapshots normalize-serialized-html
html = html5lib.serialize(
document,
tree='etree',
omit_optional_tags=False,
quote_attr_values='always',
alphabetical_attributes=True,
strip_whitespace=True,
)Alphabetical attributes and fixed quoting reduce snapshot churn. `strip_whitespace=True` can change meaningful spacing around inline nodes, so test version 1.1 output against real templates first.
Pass the HTTP charset with the byte stream pass-response-charset
from urllib.request import urlopen
import html5lib
with urlopen('https://example.com/') as response:
document = html5lib.parse(
response,
transport_encoding=response.info().get_content_charset(),
)Keep the body undecoded so html5lib can apply transport, byte-order, metadata, and fallback signals in the intended order. A predecoded string bypasses that 1.1 encoding path.
Use html5lib behind Beautiful Soup use-beautifulsoup-parser
from bs4 import BeautifulSoup
soup = BeautifulSoup(broken_markup, 'html5lib')
for link in soup.select('main a[href]'):
print(link.get('href'))This retains Beautiful Soup selectors while changing only its tree builder. Compare the malformed page under `html5lib` and `lxml` before applying the slower backend to an entire crawl.
Remove script and style bodies before text extraction extract-visible-text
import html5lib
tree = html5lib.parse(markup, namespaceHTMLElements=False)
for tag in ('script', 'style'):
for node in tree.iter(tag):
node.clear()
text = ' '.join(part.strip() for part in tree.itertext() if part.strip())ElementTree `itertext()` includes JavaScript and CSS text. Clear those 2 element types before joining content intended for indexing or analysis.
Sanitize untrusted HTML with a maintained package sanitize-with-nh3
import nh3
clean = nh3.clean(
'<script>alert(1)</script><p><b>safe</b></p>',
tags={'p', 'b'},
)
print(clean)html5lib 1.1 deprecates its own sanitizer. A successfully parsed tree can still contain executable URLs, event attributes, and unsafe elements, so use a maintained allowlist sanitizer.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| beautifulsoup4 | PyPI | Use it for a friendlier selection and traversal API, choosing html5lib or another parser behind it per workload. |
| lxml | PyPI | Use it for fast recovery plus XPath and CSS selection when matching browser repair exactly is unnecessary. |
| selectolax | PyPI | Use it for high-volume HTML extraction where native parsing speed matters more than the html5lib tree algorithm. |
More data guides
numpy · fsspec · pandas · sqlalchemy · pyarrow · lxml · the whole shelf →
How this guide is made: grounded in the library's documentation, release notes, changelog, and issue history, on a fixed rubric — not a hands-on install of every release. The 50 most-downloaded entries are additionally install-verified in clean containers. Corrections: contact the desk.