Scrapy earns its weight when a crawl becomes a system
A Scrapy spider can follow pagination, extract fields with CSS or XPath selectors, and yield records while the engine keeps other requests moving. That asynchronous scheduler is the reason to choose it over a loop around an HTTP client. Failed requests do not have to stop unrelated work, and settings can limit concurrency per domain, add a download delay, or let AutoThrottle adjust the pace.
The framework also owns the less glamorous parts that accumulate around a real crawler. It handles cookies, authentication, compression, caching, robots.txt, depth limits, and feed exports. JSON, CSV, and XML output can go to local storage, FTP, or S3, while item pipelines provide a place to clean or persist each record. With 96,670 lines of source across 710 files in our checkout, this is infrastructure rather than a small scraping helper.
That scope pays off when a crawl must run repeatedly and leave an auditable result. The interactive shell lets you test selectors against a response before editing a spider. Built-in statistics, pause and resume support, signals, middleware, and extensions give operators places to observe or alter behavior. A script that fetches 10 known pages probably does not need any of this. A crawler that discovers thousands of URLs often does.
Python 3.11 and the reactor model are firm boundaries
The current README requires Python 3.11 or newer. Installation can be one pip command, but the guide points out that lxml, cryptography, and pyOpenSSL may need native libraries. On Debian and Ubuntu it lists development headers for XML, XSLT, zlib, FFI, and SSL. Windows users are steered toward conda-forge because a pip installation may require Microsoft C++ Build Tools.
Scrapy still rests on Twisted even though new projects enable asyncio by default. Its documentation warns that importing twisted.internet.reactor can install a reactor as a side effect, after which the process cannot switch to a different one. There are helpers for crossing between Deferred objects and asyncio futures, but an application with an established event-loop policy needs deliberate integration. The reactorless path introduced in Scrapy 2.15 remains labeled experimental.
Scrapy 2.19.0 added an experimental aiohttp download handler that becomes the default without a reactor. The same release added RemoteControl, which lets a separate official MCP server inspect and control a running crawl over HTTP. That can expose crawl statistics, scheduler state, settings, and even runtime changes. It is useful for diagnosis, but enabling control over a live crawl also deserves the same access review as any other operational endpoint.
What happened when we ran it
Our sandbox installed commit e49bc32 in 31 seconds. It pulled 70 packages and occupied 100 MB on disk, then completed the build in 6 seconds. The unprivileged Debian container had 3 CPUs, 8 GB of RAM, Python 3.12, and no secrets. Pip-audit reported 0 known vulnerabilities. The repository included 7 CI workflow files and a tests directory, but no Dockerfile.
Pytest did not finish cleanly. After 432 seconds, it reported 2,673 passed, 7 failed, 50 skipped, 21 xfailed, and 76 collection/setup errors out of 2,756 tests. All 7 named failures were in tests/test_mail.py. Their shared error said is_reactorless() was called without an installed reactor or a running asyncio loop.
The log tail also showed unknown pytest.mark.asyncio warnings while collecting tests/test_utils_signal.py and tests/test_zz_resources.py. It does not establish why the marker was unknown, nor does it explain every setup error. We cannot turn that output into a dependency diagnosis. The useful finding is narrower: most tests passed, but commit e49bc32 did not give us a green full suite in the supplied fresh container.
JavaScript-rendered pages need another layer
Scrapy downloads responses; it does not pretend every response is the browser's final DOM. Its dynamic-content guide first recommends finding the underlying JSON or other data request in browser developer tools, then reproducing it directly. That is often the cleaner and cheaper crawl. When the content exists only after browser execution, the docs point to separate headless-browser options.
This boundary should shape the tool choice. Playwright is a better starting point when every target requires clicking, waiting on client-side state, or reading rendered layout. Crawlee for Python puts HTTP and Playwright crawlers in one library. Scrapy remains the stronger fit when request scheduling, extraction pipelines, throttling, and recoverable long runs are the center of the job.
October activity supports the maturity claim
GitHub showed 64,571 stars and 313 open issues and pull requests, and the repository was pushed on October 3, 2026. The latest release, 2.19.0, shipped on September 10. Recently updated work included engine refactoring, IPv6 proxy support, WebSocket support, and a report about resumed jobs overwriting batched feed files. The queue is large, but current pushes and discussion show maintainers are working in it.
Scrapy is the sensible default for a Python team whose scraper has grown into an operating service. Its docs tell you where the sharp edges are, and the 31-second installation was straightforward on our box. The failed suite changes the contribution decision: reproduce those reactor and collection errors in your environment before treating a local source checkout as release-ready.