mrkeyoor.com_
Tue 03 Nov 07:19 UTC
AI Toolsevaluationupdated 30 Sept 2026

firecrawl review

Firecrawl is a web data API that searches, scrapes, maps, crawls, and interacts with pages, then returns markdown, structured JSON, screenshots, or metadata. It solves the awkward work between an agent asking for web data and an application getting content it can use.

+1,586stars / 7d
Verdict

Our Firecrawl run installed 778 packages in 46 seconds, but the build failed and 170 of 2,457 tests failed, so self-hosting is for teams ready to own code integration as well as infrastructure. The hosted API is the sensible first trial when search, crawling, extraction, and browser actions belong behind one interface. Self-host only when data control or deployment policy is worth securing and operating the full stack.

We ran it

Lab card: what happened when we ran firecrawlScreenshot of firecrawl (firecrawl.dev)
Install✓ · 46s778 packages · 890 MB
Build✗ · 8s
Tests✗ · 135s2222 passed · 170 failed · 65 skipped of 2457 (vitest)
Repo1974 files~360,140 lines of source · 53.3 MB · 31 CI workflows · Dockerfile

Answers from our run

Does firecrawl build from source?

Dependencies installed in 46 seconds (778 packages), and the build failed. We cloned commit 24b827e into a clean Debian container with 3 CPUs and no project-specific setup.

Do firecrawl's tests pass?

No. The project's own test command finished in 135 seconds and reported 2222 passed, 170 failed, 65 skipped (vitest).

Who should not use firecrawl?

Self-hosters expecting the cloud product in a free container: the README says the hosted edition has additional features.

What are the alternatives to firecrawl?

Crawl4AI, Crawlee, Scrapy. Our Firecrawl run installed 778 packages in 46 seconds, but the build failed and 170 of 2,457 tests failed, so self-hosting is for teams ready to own code integration as well as infrastructure.

Setup2/5Install passed, but the build and 170 tests failed
Docs4/5Clear API examples and candid self-host warnings
Community5/5September 30 push with 89 open issues and 415 PRs
Maturity3/5Wide API surface, but our source checks did not pass

Discussed on

  1. hnShow HN: Firecrawl-Simple – Stable fork of Firecrawl optimized for self-hosting35 points
  2. hnShow HN: Llms.txt Generator – Turn websites into a text file to feed to any LLM28 points
  3. hnShow HN: Draco – A single-binary, self-hostable Firecrawl alternative in Rust15 points
  4. hnFirecrawl's Government Index8 points
  5. hnPdf-inspector: Rust lib for PDF inspection, classification, and text extraction5 points

Who it’s for

Agent developers who want search and page extraction behind one API.
Product teams that need markdown, structured JSON, screenshots, or browser actions from the same service.
Data teams that would rather buy Firecrawl's hosted browser and proxy operations than assemble them.
Infrastructure teams with a firm reason to keep scraping jobs and results inside their own environment.

Who it’s NOT for

Self-hosters expecting the cloud product in a free container: the README says the hosted edition has additional features.
Small teams seeking a single-process scraper: the source-aligned guide runs API workers, Playwright, Redis, RabbitMQ, and PostgreSQL-backed queue services.
Operators who need a secure public endpoint from the default Compose file: Firecrawl recommends USE_DB_AUTHENTICATION=false for the first run and says production auth, TLS, and network policy are your job.
Workloads that cannot tolerate silent extraction drift: open issue #4653 documents local-model input truncation producing wrong JSON without a warning.
Closed-source network services whose owners have not reviewed AGPL-3.0 obligations; the repository says only its SDKs and some UI components use MIT.

Setup reality

Our sandbox installed 778 pnpm packages in 46 seconds and used 890 MB. The build then failed after 8 seconds with TypeScript errors around an unknown PDF result and unresolved @mendable/firecrawl-rs imports. Vitest ran for 135 seconds: 2,222 tests passed, 170 failed, and 65 were skipped out of 2,457.

The source-aligned self-host guide calls for the Firecrawl API and workers, Playwright, Redis, RabbitMQ, NuQ PostgreSQL, and an optional FoundationDB queue backend. AI-backed features need OpenAI, an OpenAI-compatible endpoint, or Ollama. Hosted use needs a Firecrawl API key instead.

The default Compose path publishes port 3002 and begins without database authentication. Its Redis, RabbitMQ, and PostgreSQL services have no persistent volumes in the checked-in file. Before production, you must add authentication, TLS, network policy, backups, monitoring, capacity limits, and rollback procedures.

The hosted API is the Firecrawl most teams should try first

Firecrawl's v2 API puts search, scraping, site mapping, crawling, batch jobs, and browser interaction behind one service. A caller can ask for markdown, HTML, structured JSON, screenshots, links, images, or page metadata. That is useful when an agent must find a source, render its JavaScript, click into the right state, and return content without your team wiring a search provider to a browser farm and an HTML cleaner.

The quick start is short: create an API key and call the hosted endpoint from Python, JavaScript, curl, or the CLI. The README also lists Go, Java, Elixir, Rust, Ruby, .NET, and PHP SDKs. An MCP server and a Claude Code compatible skill bring the same web access into agent clients. Hosted use is the sensible evaluation path because it tests extraction on your difficult pages before you commit to running Firecrawl's infrastructure.

What happened when we ran it

Our run cloned commit 24b827e into an unprivileged Debian container with 3 CPUs, 8 GB of RAM, Node.js 22, and no secrets. The repository contained 1,974 files, about 360,140 lines of source, and occupied 53.3 MB. From apps/api, pnpm installed 778 packages in 46 seconds and left 890 MB on disk.

The build failed with exit code 2 after 8 seconds. TypeScript treated pdfResult as unknown in several PDF-engine lines. It also could not resolve @mendable/firecrawl-rs in the Playwright engine and helpers that extract attributes, images, links, and metadata or remove unwanted elements. The log establishes those compiler errors. It does not say whether the Rust-backed package needed another build step, a generated artifact, or a different dependency path.

Vitest exited 1 after 135 seconds. Of 2,457 tests, 2,222 passed, 170 failed, and 65 were skipped. The file summary was 158 failed, 160 passed, and 6 skipped. The final failures came from authenticated v1 end-to-end cases for TLS verification, base64-image removal, waits, screenshots, clicks, writing, key presses, and scrolling. Those names show the affected surface, but the log tail does not prove one shared cause.

The checkout had 31 CI workflow files, a Dockerfile, and monorepo workspaces. Our scan found no top-level tests directory, although the 2,457-test run clearly exercised suites inside the project. A successful 46-second install shows the package graph can resolve in a clean container. The failed compiler step and 170 failed tests mean commit 24b827e did not clear a clean source gate on our box.

Port 3002 opens onto a stateful stack

The source-aligned guide says Compose publishes only the API on port 3002 by default. Behind it sit Firecrawl workers, Playwright, Redis, RabbitMQ, NuQ PostgreSQL, and FoundationDB for an optional queue backend. AI-backed features need OpenAI, a compatible endpoint, or Ollama. The guide recommends keeping USE_DB_AUTHENTICATION=false for the first successful scrape, which is practical for a trusted laptop network and unsafe as a public deployment design.

The checked-in Compose file gives Redis, RabbitMQ, and NuQ PostgreSQL no persistent volumes. Firecrawl tells operators to add authentication, TLS termination, network policy, backups, recovery tests, monitoring, resource limits, scaling triggers, and rollback procedures. That candor is good documentation. It also settles the buying question: the self-host option gives you source and infrastructure control, while transferring availability, job durability, upgrades, capacity, retention, and compliance to your team.

Issue 4653 makes output verification part of the job

Open issue #4653 reports that self-hosted JSON extraction can send a full page to a local model with an 8k to 32k token context, causing Ollama to truncate the instructions and return confident wrong values without a caller warning. Issue #4773 documents a separate Playwright gap: a redirected page's content can be returned without the browser's observed final URL. These are user reports, not failures from our sandbox, but both matter when provenance is part of the product.

Issue #4348 describes an anti-bot waterfall that can repeat until the job timeout on blocked URLs. A reported workaround forces one engine or excludes the URL. Together, the 3 issues argue for application-side checks: retain the requested URL, record the returned URL and source text, reject implausible empty results, bound job time, and compare extracted fields with evidence. Firecrawl reduces the browser work. It cannot decide what level of evidence your application needs.

A June release and a September push tell different parts of the health story

GitHub showed 187,053 stars, 89 open issues, and 415 open pull requests on September 30, 2026. The repository was pushed that day. Its latest tagged release was v2.11.0 from June 19, so the tag alone understates current work. Recent issue updates and same-day pushes show active maintenance, while the large pull request queue tells adopters to pin an exact revision and test it rather than assuming main and a published image move together.

Firecrawl is easiest to recommend as a hosted boundary around an ugly problem. Its API covers enough search, crawl, extraction, and interaction work to save a product team from building several services. The self-hosted edition needs a platform owner and more work than an afternoon with Docker Compose. Our 8-second build failure and 170 failed tests are the release gate to repeat on the exact tag you plan to run.

Alternatives

ProjectWhat it isPick it when
Crawl4AI gh↗A Python crawler that turns websites into content shaped for AI applications.pick this instead when you want a Python-first local crawler and will build the surrounding service yourself.
Crawlee gh↗A JavaScript and TypeScript library for browser automation and custom crawlers.pick this instead when crawler behavior belongs in your application code rather than behind a ready-made web data API.
Scrapy gh↗A Python framework for custom crawling, extraction pipelines, and request scheduling.pick this instead when you need precise crawler logic and structured item pipelines more than agent-ready markdown.

What people are saying

  1. [github-trending] firecrawl/firecrawl
  2. [producthunt] Alexandria by Firecrawl
  3. [github-trending] firecrawl/pdf-inspector

Sources

  1. Firecrawl README
  2. Firecrawl source-aligned self-hosting guide
  3. Firecrawl contributing guide
  4. Firecrawl v2.11.0 release
  5. Issue 4653: local-model extraction can truncate instructions
  6. Issue 4773: self-hosted Playwright omits the final URL
  7. Issue 4348: anti-bot waterfall can loop until timeout

More ai tools reviews

kornia · skills · anarlog · pr-agent · agent-sprite-forge · KiroCrew · the whole board →

Related reading