Scrapy earns its weight once a crawl becomes a system
A first scraper can be a request followed by a CSS selector. Scrapy becomes useful when that script needs a queue, concurrent downloads, retries, duplicate filtering, per-domain limits, item validation, and several output destinations. A spider yields requests and records while the engine coordinates the rest. That split keeps target-specific parsing separate from the machinery that decides what to fetch next and what to do when a request fails.
The codebase is substantial but contained. Our commit 11079cb checkout had 676 files, about 89,249 lines of source, and occupied 5.2 MB. Scrapy uses Twisted for event-driven networking, with current documentation also covering asyncio-oriented runners and an option to disable the Twisted reactor. Teams already running an async web service or task system should read those integration rules before starting crawls inside an existing process.
The engine handles queues while spiders own extraction
Scrapy's data flow has 9 documented stages. Initial requests move from a spider to the engine and scheduler, through downloader middleware to the downloader, then responses return through middleware to the spider. New requests loop back into the scheduler, while extracted items enter pipelines for cleaning, validation, or storage. That arrangement makes cookies, caching, proxies, retries, and instrumentation reusable across spiders instead of copied into each callback.
For output, the framework can export JSON, CSV, or XML to local files, FTP, or Amazon S3. CSS and XPath selectors cover HTML and XML, and an interactive shell helps test expressions against a response. Built-in controls include request delay, concurrent-request limits per domain, robots.txt handling, crawl-depth restrictions, HTTP caching, sessions, and AutoThrottle. These are practical features, but sensible defaults still depend on the target site's rules and capacity.
What happened when we ran it
Our sandbox installed 67 Python packages in 36 seconds and used 97 MB on disk. The build completed successfully in 11 seconds. Pip-audit found 0 known vulnerabilities in the installed dependency set. We tested commit 11079cb in a fresh Python 3.12 Debian container with 3 CPUs, 8 GB of RAM, no secrets, and no preconfigured external services.
The full test command failed after 303 seconds. Pytest reported 2,460 passed, 10 failed, 49 skipped, 17 expected failures, and 71 errors. The measurement block records 2,541 collected or set-up cases, while pytest's final timing line showed 287.01 seconds of test execution. A high passing count shows broad exercised behavior, but exit code 1 remains the outcome.
The log tail repeatedly says pytest.mark.asyncio is unknown and names collection errors across spider middleware, stats, async generators, asyncio helpers, console utilities, decorators, deferred helpers, reactors, signals, and resource tests. It does not prove whether a dependency, plugin configuration, or another condition caused the marker error. Our result therefore says this checkout did not pass in the stated sandbox, not that Scrapy's async runtime is broken.
JavaScript-heavy sites turn into request analysis or browser work
Scrapy downloads responses; it does not behave like a graphical browser by default. Its dynamic-content guide recommends finding the request that supplies the desired data, then reproducing its URL, method, headers, form fields, or body. Request.from_curl can translate a browser-exported curl command into a Scrapy request. This route usually transfers less data and parses more predictable structures than rendering a complete page.
When the data exists only in a live DOM, the same guide points to a headless browser. Playwright can be integrated through an asyncio reactor, and scrapy-playwright exists as a separate plugin. That boundary matters for project estimates. If 8 out of 10 targets require login flows, clicks, consent dialogs, and browser fingerprint work, browser automation is the core system and Scrapy may become an extra scheduling layer rather than the simplest foundation.
Version 2.18.0 improves HTTP support, not target stability
Release 2.18.0 arrived on August 20, 2026. It moved HttpxDownloadHandler to httpx2, removed the experimental label from the Twisted HTTP/2 handler, made Brotli and Zstandard support always available, added middleware for detecting copied request metadata, and changed late crawler attributes to raise RuntimeError before a crawl starts. These changes show active work on protocol handling and clearer failure behavior.
No crawler release can stabilize another company's markup, API, rate limits, or access policy. Selectors that pass today can return empty data after a frontend deploy. Production spiders need sample fixtures, field-level validation, response-status monitoring, and alerts for sudden record-count changes. Store enough response evidence to debug a parser without immediately recrawling the target, while observing the site's terms and avoiding retention of data you should not keep.
Current maintenance is strong, while the issue count needs context
GitHub showed 64,048 stars, 453 combined issues and pull requests, and a last push on August 26, 2026. Pull requests were active on August 27 for domain matching and disabled form controls. The repository also had 8 CI workflow files and a tests directory, with no Dockerfile in our scan. The combined open count includes feature work and pull requests, so it should not be presented as 453 bugs.
Scrapy remains the default serious choice for Python HTTP crawling because its abstractions match the work that appears after the first script succeeds. The 97 MB installed size and 11-second build are modest; learning the engine and operating target-specific failure detection are the larger costs. Adopt it when those shared controls will serve several spiders, and keep browser rendering outside the hot path unless the target leaves no cleaner source to request.