mrkeyoor.com_
Thu 17 Sept 20:44 UTC
AI Toolsevaluationupdated 27 Aug 2026

firecrawl review

Firecrawl is a web data API that can search, scrape, map, crawl, and interact with pages, then return markdown, structured JSON, screenshots, or page metadata. It gives agents and applications one interface for web work that may require JavaScript rendering or browser actions before extraction.

+2,089stars / 7d
Verdict

Our Firecrawl build failed in 9 seconds on an unresolved Rust-backed module, and 183 of 2,166 tests failed, so self-hosters should treat adoption as a platform project. The hosted API is the sensible first trial for teams that want search, crawling, extraction, and browser actions behind one interface. Self-host only when data control or deployment policy outweighs the work of securing and operating its stateful services.

We ran it

Lab card: what happened when we ran firecrawlScreenshot of firecrawl (firecrawl.dev)
Install✓ · 66s727 packages · 692 MB
Build✗ · 9s
Tests✗ · 256s1930 passed · 183 failed · 53 skipped of 2166 (vitest)
Repo1764 files~304,255 lines of source · 51.3 MB · 31 CI workflows · Dockerfile

Answers from our run

Does firecrawl build from source?

Dependencies installed in 66 seconds (727 packages), and the build failed. We cloned commit 9835405 into a clean Debian container with 3 CPUs and no project-specific setup.

Do firecrawl's tests pass?

Not all of them: 1930 of 2166 passed and 183 failed when we ran the project's own test command (vitest). Some failures need services or credentials a bare container does not have.

Who should not use firecrawl?

Developers expecting a small local scraper: the self-host stack runs an API, workers, Playwright, Redis, RabbitMQ, and PostgreSQL.

What are the alternatives to firecrawl?

Crawl4AI, Scrapy, Browserless. Our Firecrawl build failed in 9 seconds on an unresolved Rust-backed module, and 183 of 2,166 tests failed, so self-hosters should treat adoption as a platform project.

Setup2/5Install passed, but the build and 183 tests failed
Docs4/5Clear API examples and candid self-host production warnings
Community5/5172,982 stars, current pushes, and recently updated issues
Maturity4/5Wide API and SDK coverage, with source integration failures

Discussed on

  1. hnShow HN: Firecrawl-Simple – Stable fork of Firecrawl optimized for self-hosting35 points
  2. hnShow HN: Llms.txt Generator – Turn websites into a text file to feed to any LLM28 points
  3. hnShow HN: Draco – A single-binary, self-hostable Firecrawl alternative in Rust15 points
  4. hnPdf-inspector: Rust lib for PDF inspection, classification, and text extraction5 points
  5. hnFast Rust Library for PDF text extraction5 points

Who it’s for

Agent developers who need current web pages in model-ready formats.
Data teams collecting pages through crawl, batch, and asynchronous job APIs.
Applications that need search, screenshots, structured extraction, or browser interaction from one client.
Teams willing to use the hosted API or operate browsers, queues, workers, and storage themselves.

Who it’s NOT for

Developers expecting a small local scraper: the self-host stack runs an API, workers, Playwright, Redis, RabbitMQ, and PostgreSQL.
Self-hosters who need a production-ready default: the checked-in baseline has authentication disabled and defines no persistent volumes for Redis, RabbitMQ, or PostgreSQL.
Teams that require a clean source build from a fresh Node container: our commit 9835405 build stopped on an unresolved @mendable/firecrawl-rs module, and 183 tests failed.
Closed-source network service operators who have not reviewed AGPL-3.0 obligations; the README says MIT applies only to the SDKs and some UI components.
Pipelines that cannot tolerate silent or corrupt output: open issue 4375 reports self-hosted search returning success with an empty result after an internal crash, and issue 4047 reports damaged text on non-UTF-8 pages.
Buyers assuming the repository matches the hosted product feature for feature: the README explicitly says the cloud edition includes additional capabilities.

Setup reality

Our sandbox install succeeded in 66 seconds, adding 727 packages and using 692 MB. The build failed after 9 seconds because TypeScript could not resolve @mendable/firecrawl-rs in sitemap and scraper modules. Tests ran for 256 seconds: 1,930 passed, 183 failed, and 53 were skipped out of 2,166.

Hosted use starts with a Firecrawl API key. Self-hosting needs the API and workers, Playwright, Redis, RabbitMQ, and NuQ PostgreSQL. AI-backed features need OpenAI, a compatible endpoint, or Ollama. The default Compose API has authentication off, and its three state services have no persistent volumes.

Development uses Node.js 22 and pnpm 11.4.0, with Redis running separately while the harness manages PostgreSQL and RabbitMQ. The repository has a Dockerfile and 31 CI workflows, but our clean source build did not obtain or produce the Rust-backed package needed by the TypeScript imports.

One API covers search, scraping, crawling, and browser actions

Firecrawl puts five related jobs behind one service: web search, single-page scraping, URL mapping, site crawling, and browser interaction. A response can contain markdown, HTML, screenshots, structured JSON, links, images, or page metadata. That common shape is useful for an agent that must find a source and then extract it without switching between a search provider, browser runner, and HTML cleaner.

The hosted API is the shortest route into the product. Its README shows Python, JavaScript, Go, Java, Rust, Ruby, .NET, PHP, and Elixir clients, plus a CLI and an MCP server. A Claude Code skill is also available. Self-hosting serves a different buyer: someone willing to own the browser fleet, queues, database, network egress, and upgrades in exchange for infrastructure control.

What happened when we ran it

Our run used commit 9835405 in a fresh unprivileged Debian container with 3 CPUs, 8 GB of RAM, Node.js 22, and no secrets. The checkout contained 1,764 files, about 304,255 lines of source, and 51.3 MB before installation. From apps/api, pnpm installed 727 packages in 66 seconds and left 692 MB on disk.

The build failed with exit code 2 after 9 seconds. TypeScript could not resolve @mendable/firecrawl-rs or its type declarations. The same missing import appeared in sitemap handling and scraper code for documents, Fire-engine, PDFs, Playwright, attributes, images, links, metadata, and element removal. The log does not say whether that package needed generation, a separate download, or a different build entry point, so we cannot assign a cause beyond the unresolved module.

Vitest ran for 256 seconds and exited 1. Of 2,166 tests, 1,930 passed, 183 failed, and 53 were skipped. The summary counted 90 failed test files, 138 passed files, and 5 skipped files. Its final failures covered authenticated v0 search and crawl-status routes, crawl cancellation, and LLM extraction. One test reached its 120-second timeout. Those lines show several failing areas, but they do not establish one shared cause.

The repository has 31 CI workflow files, a Dockerfile, and monorepo workspaces. Our scan found no top-level tests directory, although the 2,166-test run plainly exercised a large suite under apps/api. A successful 66-second install is encouraging for contributors. The failed compiler step and 183 failed tests still mean this exact checkout did not pass clean-room development gates.

Hosted use avoids the self-host operating burden

A hosted trial needs a Firecrawl API key and a client call. Search can return page content with results, while crawl and batch operations use job IDs that the SDKs poll. Browser interaction can click, type, wait, scroll, and expose a CDP connection. This is enough surface area to test extraction quality on your own troublesome pages before making any infrastructure decision.

Release v2.11.0, published June 19, 2026, added PII redaction, a deterministic JSON format, broader video discovery, CDP access, and monitor changes. The README also says the cloud service contains features beyond the open-source edition. Check the exact endpoint you need against the self-host documentation before treating a hosted example as a promise about the community deployment.

The default Compose stack needs security and persistence work

The checked-in Compose baseline runs the Firecrawl API and workers, Playwright, Redis, RabbitMQ, and NuQ PostgreSQL. FoundationDB is an optional queue backend. Only port 3002 is published by default, and the guide recommends keeping the NuQ PostgreSQL queue until an operator intentionally chooses FoundationDB. AI-dependent work needs OpenAI, an OpenAI-compatible endpoint, or Ollama. Proxy and search-provider settings are separate choices.

The guide is unusually direct about what remains unfinished for production. USE_DB_AUTHENTICATION=false is the recommended first-run setting, not a secure public deployment. Redis, RabbitMQ, and NuQ PostgreSQL have no persistent volumes in the root Compose file. Operators must add authentication, TLS, network policy, backups, recovery tests, monitoring, resource limits, scaling rules, and rollback procedures before exposing the API or trusting jobs to survive service replacement.

Two open issues show why callers must verify output

Issue 4375 documents a self-hosted /v2/search path that can hit a malformed DuckDuckGo result URL, log an internal error, then answer HTTP 200 with success: true and an empty result. A caller cannot distinguish that failure from a legitimate search with no matches. The report says using a SearXNG endpoint avoids that path, but applications should still alert on suspicious empty responses instead of treating the status field as sufficient evidence.

Issue 4047 reports Fire-engine decoding every response as UTF-8. Pages served as Shift_JIS, EUC-KR, GBK, Windows-1251, or ISO-8859 can therefore reach markdown with damaged characters. Issue 4393 describes a different class of drift: onlyCleanContent exists in server schemas but was absent from 9 SDKs and the OpenAPI description, allowing the Python client to discard the option without warning. Multilingual or schema-sensitive pipelines need fixtures that compare requested options with returned content.

August activity is current despite the June release

GitHub recorded 172,982 stars, 75 open issues, and 555 open issues and pull requests combined when we fetched the repository on August 27, 2026. The last push was also August 27. Issue 4426 was opened that day, and issue 3782 received an update, so current activity is visible in both code and issue work. Release v2.11.0 is older than the latest push, which is not evidence of abandonment.

Firecrawl earns a hosted trial when one API can replace several web-data components. The 727-package install completed, but the 9-second build failure and 183 failed tests make the source route a conditional choice. Require a clean build of the exact tag, run the relevant E2E routes with your providers, and test persistence and authentication before a self-hosted instance carries production traffic.

Alternatives

ProjectWhat it isPick it when
Crawl4AI gh↗A Python crawler focused on turning web pages into input for AI applications.pick this instead when you want a Python-first local crawler and can build your own service boundary.
Scrapy gh↗A mature Python framework for custom spiders, item pipelines, throttling, and exports.pick this instead when precise crawl rules and data pipelines matter more than agent-oriented markdown.
BrowserlessA hosted or self-run browser service that exposes common automation protocols.pick this instead when browser sessions are the main requirement and you will write the extraction layer yourself.

What people are saying

  1. [github-trending] firecrawl/firecrawl
  2. [producthunt] The new Firecrawl MCP
  3. [github-trending] firecrawl/pdf-inspector

Sources

  1. Firecrawl README
  2. Firecrawl source-aligned self-hosting guide
  3. Firecrawl contributing guide
  4. Firecrawl v2.11.0 release
  5. Issue 4375: self-hosted search reports a crash as an empty success
  6. Issue 4047: non-UTF-8 pages are decoded incorrectly
  7. Issue 4393: onlyCleanContent missing from SDKs

More ai tools reviews

rowboat · skills · superpowers-zh · yolov5 · eve · MemOS · the whole board →