Browser control is the reason to choose Crawl4AI
Crawl4AI wraps Playwright with output meant for retrieval and agent systems. A crawl can wait for dynamic content, execute JavaScript, preserve sessions, use authenticated browser profiles, rotate proxies, capture screenshots, scan a full page, and follow links through breadth-first or other deep-crawl strategies. Results can include raw and filtered Markdown, media, metadata, links, and structured JSON. This is much more control than an article-text parser provides.
Extraction can stay deterministic through CSS and XPath schemas, or add heuristic filtering, BM25 relevance, chunking, embeddings, and model calls. That choice matters because many product pages have repeatable DOM structures and do not justify sending content to a language model. The asynchronous API is the main path. The README marks its synchronous Selenium variant as deprecated, so new code should not build around the old interface.
The 5,368 MB install is larger than many crawler servers
Our environment installed 158 packages in 141 seconds and occupied 5,368 MB. The checked-out repository itself was only 28.9 MB, with 916 files and about 167,178 source lines at commit 7e80152. Browser binaries and Python ML or parsing dependencies can make a crawler much larger than its source. Container registries, CI caches, autoscaling nodes, and developer laptops all pay that storage and transfer cost.
The project offers a base package and optional extras for torch, transformers, cosine similarity, PDF support, and the deprecated sync mode. Installing everything for convenience can pull capabilities a service never uses. Start with the exact crawl and extraction strategy required, then add extras deliberately. Pip-audit found 0 known vulnerabilities in our installed packages, which is useful, though it does not assess browser binaries, container configuration, or application-level request handling.
What happened when we ran it
The source build exited 1 after 12 seconds. Its supplied tail contains SyntaxWarnings for invalid escape sequences in 3 documentation example files. Those warnings may be related to the failure or merely the last visible output; the log tail does not contain the terminating exception, so naming a cause would be guesswork. The factual result is that commit 7e80152 installed but did not complete the configured build in our Python 3.12 Debian container.
Pytest exited 3 after 69 seconds. The final report shows 492 warnings and 35 collection or setup errors, followed by an internal SystemExit: 1 and an Event loop is closed warning. A separate summary inside that log says RESULTS: 84 passed, 2 failed out of 86 tests. Those lines describe different stages of a broken run and should not be collapsed into a pass rate. The top-level measured result remains 0 collected passes, 0 collected failures, and 35 errors for the command.
Docker security improved, while the quickstart has a reachability trap
Version 0.9.0 changed the Docker API to require authentication by default and treat incoming request bodies as untrusted. The README says an instance without a token binds to loopback. Issue 2090 explains the operational consequence: the advertised docker run -p 11235:11235 command sets no token, so the server binds 127.0.0.1 inside the container and host requests fail even though the container can appear healthy.
A production setup needs a token, an intentional bind address, browser shared memory, resource limits, and logs that do not leak cookies or credentials. Issue 2091 also reports Compose v5 rejecting the provided file because two process-limit forms conflict. These reports concern v0.9.2 deployments, not every way to import the Python library. They are enough reason to test the exact container command from a clean host rather than copying a quickstart into automation.
Browser cleanup needs a multi-day soak test
Release v0.9.2 says it fixed a task and page leak when a streaming crawl closes. Issue 2202 reports another lifecycle problem on non-streaming /crawl: after 29 days and only a few jobs per day, a container held 46 Chromium processes, used 957 MiB, and consumed about 31% of one CPU core while idle. The report ties process start times to earlier crawls and describes a restart as recovery.
That is one deployment report, but browser leaks are expensive enough to test directly. Run a representative crawl mix for several days, include navigation failures and timeouts, then graph page, context, renderer, file descriptor, memory, and CPU counts after traffic stops. Issue 2155 adds a related failure path where Playwright driver processes survive if browser startup fails. A health endpoint alone will not reveal either accumulation pattern.
PDF and content filters can return misleading results
The default Docker image does not include pypdf, according to issue 2127, so its allowed PDF strategy fails at runtime unless the broad all extra is installed. Issue 2135 reports a separate PDF path where extraction succeeds but anti-bot detection marks the result failed because placeholder HTML is only 33 bytes. Downstream code that checks success can discard valid PDF Markdown.
Content filtering also needs fixtures. Issue 2110 shows filtered Markdown collapsing whitespace inside syntax-highlighted code, while raw Markdown remains correct. Issue 2161 reports overlay removal deleting the entire body when a theme class contains popup. GitHub showed 79,482 stars, 164 combined open issues and pull requests, and a last push on August 25, 2026. The project is active and capable. Its current edge cases make output comparison, process monitoring, and pinned deployment tests mandatory rather than optional polish.