mrkeyoor.com_
Sat 31 Oct 05:56 UTC
Automationevaluationupdated 03 Oct 2026

airflow review

Apache Airflow schedules and monitors workflows whose steps and dependencies are written in Python. It gives recurring, mostly stable jobs a shared scheduler, run history, retries, and a web interface, replacing scattered cron entries with workflow code a team can review.

+52stars / 7d
Verdict

Our Airflow run installed 152 packages in 94 seconds and built in 21 seconds, but pytest stopped after 7 seconds on an unrecognized option, so the source tree expects more preparation than the package quick start suggests. Use Airflow when scheduled Python DAGs, backfills, provider integrations, and years of run history justify shared infrastructure. Pick a smaller orchestrator for a young pipeline estate, and use a stream processor for continuous events.

We ran it

Lab card: what happened when we ran airflowScreenshot of airflow (airflow.apache.org)
Install✓ · 94s152 packages · 196 MB
Build✓ · 21s
Tests✗ · 7sran, no count parsed
Known vulns0(pip-audit)
Repo14427 files~1,976,757 lines of source · 193 MB · 56 CI workflows · Dockerfile

Answers from our run

Does airflow build from source?

Dependencies installed in 94 seconds (152 packages), and the build succeeded in 21 seconds. We cloned commit 413ae46 into a clean Debian container with 3 CPUs and no project-specific setup.

Do airflow's tests pass?

No. The project's own test command exited unsuccessfully after 7 seconds, and its output did not expose an individual test count.

Does airflow have known vulnerabilities in its dependencies?

pip-audit found none in the dependency tree at the time of our run.

Who should not use airflow?

Teams processing continuous event streams: the README says Airflow is not a streaming system and recommends consuming stream data in batches.

What are the alternatives to airflow?

Prefect, Dagster, Argo Workflows. Our Airflow run installed 152 packages in 94 seconds and built in 21 seconds, but pytest stopped after 7 seconds on an unrecognized option, so the source tree expects more preparation than the package quick start suggests.

Setup2/5Install and build passed, but tests stopped and production needs services
Docs5/5Clear limits, version matrix, constraints, tutorials, and deploy guides
Community5/547,036 stars, an October 3 push, and active issue work
Maturity5/5Versioned core, provider releases, migrations, and long-lived operations

Discussed on

  1. hnApache Airflow: programmatically author, schedule, monitor workflows6 points

Who it’s for

Data and platform teams operating scheduled pipelines across databases, cloud services, and compute systems.
Python teams that want workflow definitions reviewed and versioned with application code.
Operators who need backfills, retries, run history, and several execution choices for recurring jobs.
Organizations prepared to run a metadata database and treat the scheduler as production infrastructure.

Who it’s NOT for

Teams processing continuous event streams: the README says Airflow is not a streaming system and recommends consuming stream data in batches.
Workflows whose graph changes radically on every run: Airflow says it works best when DAG structure is mostly static and slowly changing.
Jobs that pass large payloads between steps: the project advises using external services for heavy data and reserving XCom for metadata.
Windows-only operators: native Windows is unsupported, WSL2 or Linux containers are the documented paths, and Linux is the supported production platform.
Small teams wanting a low-maintenance cron replacement: our source install pulled 152 packages, while production also needs a supported database, an executor, and managed connections.

Setup reality

Our sandbox run at commit 413ae46 installed 152 packages in 94 seconds and occupied 196 MB. The build passed in 21 seconds, but tests stopped after 7 seconds with exit 4 because pytest did not recognize --asyncio-mode=strict. The log does not establish why. Pip-audit found 0 known vulnerabilities.

A useful deployment needs a supported metadata database, an executor, logs, and credentials for every external system a DAG touches. Airflow publishes versioned constraints because its README warns that an unconstrained pip install can sometimes produce an unusable environment.

Linux is the supported production platform. SQLite is limited to local development and tests, while Windows users need WSL2 or Linux containers. The 193 MB checkout contained 14,427 files and 56 CI workflow files plus a Dockerfile; our scan found no top-level tests directory.

Airflow 3.3.2 is for scheduled graphs that stay recognizable

Airflow 3.3.2 turns a Python definition into a scheduled graph of tasks, then records each run in a metadata database. Its scheduler decides when work is ready, an executor sends it to workers, and the web interface shows dependencies, failures, retries, and backfills. That split pays off when a pipeline must be understood by people other than its first author. A DAG can live beside ordinary code, pass through review, and retain an operational history that cron does not provide.

The boundary is unusually clear for a project with 14,427 files. Airflow says it works best when a DAG is mostly static and changes slowly between runs. It is not a streaming engine, even if teams use it to pull records from streams in batches. Tasks should be idempotent, and XCom is for metadata rather than large datasets. If every run invents a new graph or moves a large payload between steps, you are working against the design.

What happened when we ran it

Our sandbox run at commit 413ae46 installed 152 packages in 94 seconds and occupied 196 MB on disk. The checkout itself was 193 MB, with about 1,976,757 lines of source. On the same unprivileged Debian container with 3 CPUs and 8 GB of RAM, the build completed in 21 seconds. Pip-audit reported 0 known vulnerabilities for the exact environment we measured. Those are repository mechanics, not scheduler throughput or task performance.

The test step failed after 7 seconds with exit 4. Pytest rejected --asyncio-mode=strict, naming /work/repo/pyproject.toml as its configuration file and /work/repo as the root. The log tail does not identify why that option was unavailable, so blaming a missing plugin or a dependency mismatch would be guesswork. The useful finding is narrower: a successful 94-second install and 21-second build did not make the configured test command runnable in our fresh container.

Our scan counted 56 CI workflow files and found a Dockerfile, but no top-level tests directory. That last signal does not mean a 1,976,757-line repository has no tests. Airflow spreads core code, providers, clients, deployment assets, and development tooling across a large source tree. A contributor should use the documented development environment instead of assuming the PyPI installation path is also the source-development path.

Airflow 3.3.2 expects a versioned constraint file

The 3.3.2 README warns that plain pip install apache-airflow may sometimes fail or produce an unusable installation. Airflow keeps library dependencies open in pyproject.toml, then publishes known-working constraint files for each Airflow and Python version. Only pip and uv are officially supported. Poetry and pip-tools users are told to convert the constraints to their own workflow, which puts compatibility work back on the adopting team.

Our 152-package install succeeded, but production begins after package resolution. You still choose a metadata database and executor, configure logs, store credentials, and install providers for the systems each DAG calls. Core Airflow and provider packages have separate release tracks. That flexibility is why Airflow can coordinate a broad data estate, and it is also why casual upgrades are a poor operating habit. Pin the environment, test DAGs and migrations, then move every component together.

Linux and a real database are production requirements

Airflow 3.3.2 supports Python 3.10 through 3.14. Its matrix lists PostgreSQL 14 through 18 and MySQL 8.0 or 8.4, while MariaDB is not recommended. SQLite 3.15.0 or newer is acceptable for local work and tests, but the README explicitly rules it out for production. Native Windows is unsupported; developers need WSL2 or Linux containers, and production execution is supported on Linux distributions.

The 3-CPU, 8 GB sandbox did not exercise a scheduler, workers, a production database, or an external service. Those parts carry the real operating cost: database backups and migrations, task logs, secrets, worker capacity, and credentials for every destination. Airflow is most useful as the coordinator of large jobs running elsewhere. Treating XCom as a data bus or the metadata database as disposable storage turns the control plane into the bottleneck.

The October 3 push shows active work and a large queue

GitHub showed 47,036 stars and a push on 2026-10-03. Its combined counter held 1,829 open issues and pull requests; separate searches returned 1,027 issues and 802 pull requests. Apache Airflow 3.3.2 shipped on 2026-09-17. Those dates rule out a dormant-project concern, while the queue size makes the tradeoff plain: this is a large platform with many integrations, release paths, and user environments to support.

The 3.3.2 release also changed authorization behavior around backfill IDs and explicit credentials, alongside a long list of scheduler, API, migration, and interface fixes. Mature does not mean motionless. Teams need to read release notes, test upgrades against their own providers, and plan database changes. Airflow's documentation earns trust here because it states platform limits, explains constraints, and separates guides for the core, provider packages, containers, and Helm deployments.

Choose Airflow when 152 packages buy shared operations

Airflow earns its place when several teams need one visible record of scheduled work, failed runs, retries, and backfills. The 94-second install and 21-second build show that the code can be assembled quickly in a clean container, while the test stop and production requirements show why adoption is an operations decision. For a few jobs, Prefect or plain scheduling may demand less care. For data assets, Dagster offers a different center of gravity. Kubernetes-first shops should compare Argo Workflows before adding another control plane.

Alternatives

ProjectWhat it isPick it when
Prefect gh↗A Python workflow orchestrator with a direct path from local functions to managed or self-hosted execution.pick this instead when developers want ordinary Python control flow and less scheduler administration for a newer workflow estate.
DagsterA data orchestrator built around software-defined assets, lineage, and developer tooling.pick this instead when data assets and their dependencies are the main abstraction your team wants to manage.
Argo Workflows gh↗A Kubernetes-native workflow engine that runs container steps as cluster resources.pick this instead when every job already belongs in Kubernetes and Python-authored DAGs would add an unwanted control plane.

Sources

  1. Apache Airflow README
  2. Apache Airflow 3.3.2 release
  3. Apache Airflow issue and pull request activity
  4. Apache Airflow documentation

More automation reviews

moli · MatrixMedia · skill-recorder · BPB-Wizard · wenzi-xiaohongshu-workflow · open-dot · the whole board →