One API covers HTTP crawlers and real browsers
Crawlee sits above several ways of retrieving a page. A lightweight crawler can use HTTP with Cheerio or JSDOM, while browser jobs can run through Playwright or Puppeteer. The surrounding system stays familiar: persistent URL queues, retries, routing, sessions, proxies, hooks, datasets, and resource-based scaling. That shared structure is the reason to choose Crawlee over calling a browser library directly. It is especially useful when some targets need JavaScript rendering and others do not.
The repository itself is a large pnpm workspace. At commit 69a824b, our checkout had 2,999 files, roughly 119,207 source lines, and occupied 223.3 MB before installation. The scan found a tests directory and 9 CI workflow files, with no Dockerfile. The README says deployment Dockerfiles are available through the wider project experience, but this measured checkout did not contain one. Plan to inspect the scaffold or deployment path you will actually use.
Browser support is optional and intentionally separate
A manual Playwright example installs both crawlee and playwright because browser automation is not bundled into the main package. That keeps an HTTP-only project from paying for browser packages and binaries. The CLI can create a starter crawler, while local queues and datasets default to ./storage. Node.js 16 or newer is required. Teams should select HTTP first when the target returns usable HTML and add a browser only for pages that need scripts, interaction, or screenshots.
Even without a bundled browser, our pnpm install used 1,255 MB and took 166 seconds. The harness reported 1 installed package for the workspace, so that count should not be read as a count of every transitive module present on disk. The build completed in 38 seconds. Browser projects will have additional downloads and operating requirements that this repository-only measurement did not include. Proxy accounts and cloud storage are also outside the local build.
What happened when we ran it
We ran commit 69a824b in an unprivileged container with 3 CPUs, 8 GB of RAM, Node 22, and no secrets. Installation succeeded in 166 seconds and the build passed in 38 seconds. The test command ran for 374 seconds before failing with exit code 1. Vitest reported 1,559 passed, 192 failed, and 38 skipped out of 1,801 tests.
The final log showed Vitest failing to start a forks worker for test/core/error_snapshotter.test.ts. The cause line says the pool runner timed out waiting for the worker to respond. The log tail does not prove why 192 tests failed or whether one resource limit explains all of them, so we will not assign a cause. It does establish that this commit did not pass the full suite in our 3-CPU, 8 GB sandbox.
Queues and storage are the feature, and the risk
Persistent requests let a crawl resume, deduplicate URLs, and enqueue discovered links. Datasets and key-value storage give handlers a standard output path. Retries, session management, and proxy rotation reduce the application code around unreliable sites. These features matter once a crawler lasts longer than one process or visits enough pages that restarts and duplicate work become expensive. A one-page extraction script may be clearer with a direct HTTP request and an HTML parser.
Crawlee's 374-second test run shows why storage and scheduling paths deserve their own acceptance suite. Test restart recovery, duplicate prevention, maximum crawl depth, oversized records, and partial output. Issue 4065 reports that v4 transactional writes can fail during commit after pushData has already returned, changing where an application can catch and repair a record. That is a v4 report, so stable v3 users should treat it as migration evidence rather than current v3 behavior.
v4 work is changing request management
Recent issues and pull requests show active changes around throttling, recoverable state, request locks, storage commits, and adaptive crawling. Issue 4039 reports a v4 path where requests supplied by a wrapped manager ignored a domain's backoff. Issue 4064 says adaptive Playwright crawling did not expose a browser choice. These are narrow reports with reproductions, and open work may change them. They still justify pinning a version and testing 429 responses, pacing, browser selection, and shutdown before upgrading.
GitHub recorded the last push on August 26, 2026, with 25,514 stars and 153 combined issues and pull requests. Stable release v3.18.1 arrived on August 12 and fixed storage purging, Cloudflare challenge handling, and crawl-depth behavior. The active maintenance is reassuring. Our 192 failed tests prevent an unconditional recommendation for this checkout. Use Crawlee for the orchestration it provides, then verify the precise crawler class, storage backend, and target behavior you plan to run.