orchestration

Apache Airflow

Apache Airflow is the long-standing open-source workflow orchestrator. This profile covers what it does, how it is operated, and where it stops being the right choice.

What the product actually does

Airflow schedules directed acyclic graphs of tasks. Each task is a Python callable, the scheduler decides when it runs, and the metadata database records every run, its state and its logs.

The maturity is the argument. There are provider packages for nearly every system that has an API, an enormous body of operational knowledge, and a managed offering from every cloud. A team adopting an orchestrator in 2026 is choosing between Airflow's depth and everything else's ergonomics.

How it is operated and what it costs

Open source and free, with the cost being a scheduler, a metadata database and a worker pool. The scheduler is a single point of failure unless it is run in high availability, which is a configuration most teams eventually need.

Managed Airflow prices per environment plus worker capacity, and the environment fee dominates at small scale. A team running twelve daily jobs pays mostly for the privilege of not operating the scheduler.

Where it breaks down

The task model. Airflow knows about tasks and their order; it does not know what data those tasks produce. When a schema changes, the DAG runs and the failure appears downstream rather than at the point of the change.

Scheduling latency is also a constraint. The default scheduler loop is measured in seconds, and a pipeline that needs sub-minute triggering is outside the design.

What switching costs

Moderate. DAGs are Python and the logic ports, but the operator usage, the connection definitions and the scheduling semantics do not.

Sources

Apache Airflow public documentation, read October 2026.