beautifulsoup4
Beautiful Soup parses HTML and XML into a Python object tree you can search and edit. It sits on top of a real parser (html.parser from the standard library, or lxml and html5lib if installed) and gives you find, find_all, CSS selectors, and tree navigation that tolerate the broken markup real websites ship. It is the extraction half of scraping; you still need requests or httpx to fetch the pages.
Still the friendliest way to pull data out of messy HTML, and the pure-Python default keeps installs painless. Use it for jobs measured in pages; move to lxml or selectolax when you measure in millions.
Use it if
- You are scraping or cleaning real-world HTML that is malformed, mis-nested, or inconsistently encoded
- You want extraction code that reads like the page structure it targets: find_all and select over regex heroics
- You need to edit markup (strip tags, rewrite attributes, pull fragments), not just read it
- You are prototyping and want results in ten lines using only the standard library parser
- Throughput matters: bs4 builds a full Python object tree, and lxml or selectolax parse the same pages far faster at bulk-scraping volume
- The page renders its content with JavaScript: no HTML parser will ever see that data; you need Playwright or the site's underlying JSON API
- You are building a crawling pipeline: Scrapy with parsel gives you XPath plus CSS with request handling, throttling, and retries built in
- You are parsing one well-formed XML feed: lxml or the standard library's ElementTree does it without an extra dependency
Setup reality
pip install beautifulsoup4 works everywhere because the default backend is Python's own html.parser: no compiled extensions, no wheel drama. Wanting speed means adding lxml, which does need wheels (fine on mainstream platforms, annoying on Alpine). Two classic footguns: the import name is bs4, not beautifulsoup4, and PyPI still hosts an ancient package literally named BeautifulSoup (version 3), so a from-memory install can hand you an abandoned codebase. Also pin your parser explicitly, because different backends build slightly different trees from bad markup.
Patterns
Parse a document and grab the titleparse-html
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "html.parser")
print(soup.title.string)
print(soup.h1)Always name the parser explicitly; leaving it out picks the best installed backend, so the tree can change between machines.
Find tags by name, class, and attributesfind-elements
links = soup.find_all("a", class_="result", limit=10)
for a in links:
print(a["href"], a.get_text(strip=True))
row = soup.find("tr", attrs={"data-id": "42"})It is class_ with a trailing underscore because class is reserved in Python; find returns None on a miss, so chain carefully.
Query with CSS selectorscss-select
cells = soup.select("table#prices tr td:nth-of-type(2)")
first = soup.select_one("div.card > h2")
if first is not None:
print(first.get_text())select is powered by the bundled soupsieve dependency and supports most modern CSS; select_one returns None rather than raising.
Get clean text out of a subtreeextract-text
text = soup.get_text(separator=" ", strip=True)
for p in soup.select("article p"):
print(p.get_text(strip=True))get_text() with no separator glues adjacent strings together into words like 'HomeAbout'; pass separator=' ' for readable output.
Read tag attributes safelyread-attributes
a = soup.find("a")
href = a.get("href") # None if missing
hard = a["href"] # KeyError if missing
classes = a.get("class", []) # class is always a listMulti-valued attributes like class come back as lists, not strings, which breaks naive == comparisons.
Move between related elementsnavigate-tree
label = soup.find("th", string="Price")
value = label.find_next_sibling("td")
section = value.find_parent("section")
for child in section.children:
print(getattr(child, "name", None))Whitespace between tags counts as text nodes, so .children and .next_sibling often return strings where you expected tags.
Scrape a live page with requestsfetch-and-parse
import requests
from bs4 import BeautifulSoup
resp = requests.get(url, headers={"User-Agent": "my-bot/1.0"}, timeout=10)
resp.raise_for_status()
soup = BeautifulSoup(resp.content, "html.parser")Pass resp.content (bytes) rather than resp.text so Beautiful Soup does its own encoding detection instead of trusting the HTTP header.
Strip tags and rewrite contentmodify-tree
for tag in soup.find_all(["script", "style"]):
tag.decompose()
h1 = soup.find("h1")
h1.string = "New title"
h1["class"] = ["headline"]
clean_html = str(soup)decompose() destroys the element permanently; use extract() instead if you want to keep the removed subtree around.
Parse only part of a big page with SoupStrainerpartial-parse
from bs4 import BeautifulSoup, SoupStrainer
only_links = SoupStrainer("a")
soup = BeautifulSoup(html, "html.parser", parse_only=only_links)
hrefs = [a.get("href") for a in soup.find_all("a")]SoupStrainer cuts memory and parse time on large pages, but it does not work with the html5lib backend.
Handle a page with wrong or missing encodingfix-encoding
soup = BeautifulSoup(raw_bytes, "html.parser", from_encoding="iso-8859-1")
print(soup.original_encoding)Only force from_encoding after checking soup.original_encoding; guessing before the auto-detection fails just adds a new bug.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| lxml | PyPI | You need speed and full XPath and can accept a compiled dependency and a stricter API. |
| selectolax | PyPI | You are CSS-selector-only and want the fastest practical HTML parsing for large crawls. |
| parsel | PyPI | You want Scrapy-style selectors (XPath plus CSS chained together) with or without Scrapy itself. |