Apache Airflow and the DAG Model of Pipeline Orchestration

How Airflow uses directed acyclic graphs to schedule, retry and backfill data pipelines, and why modelling dependencies explicitly beats chaining cron jobs together.

▶ Open the simulation

The problem with cron

Before orchestration tools became standard, a typical data team's 'pipeline' was a folder of cron jobs, each one firing at a fixed time and hoping the job before it had already finished. This worked until it didn't: a job that ran twelve minutes late because a warehouse was under load would silently corrupt the output of everything scheduled after it, and nobody would notice until an analyst asked why yesterday's dashboard numbers looked wrong. Cron encodes only time, never dependency, so it is structurally incapable of expressing 'task B should run after task A succeeds, not merely after a fixed clock offset.'

Airflow's core move was to replace time-based triggering with a dependency graph. A pipeline is expressed as a Directed Acyclic Graph, or DAG: a set of tasks (the nodes) connected by edges that mean 'this task must complete before that one starts.' Because the graph is acyclic, there is no way to accidentally create an infinite loop of dependencies — Airflow validates this at parse time and refuses to schedule a DAG that isn't a proper DAG. The scheduler's job then becomes graph traversal: walk the DAG, find tasks whose upstream dependencies have all succeeded, and hand them to a worker.

Execution dates and idempotent runs

One of Airflow's more subtle but consequential design choices is that every DAG run is tagged with a logical execution date representing the period of data it is meant to process, which is deliberately not the same as the wall-clock time the run actually executes. A DAG scheduled to run daily at 2am for '2026-07-24' processes the 24th's data even if it's kicked off late, or re-run three days after the fact — the logical date, not the clock, determines what data the task operates on. This decoupling is what makes backfilling possible: if you change a transformation and need to reprocess the last 90 days, you simply trigger 90 DAG runs, one per historical logical date, and each one queries its own date's partition rather than 'today.'

For this to work safely, tasks need to be idempotent — running the same logical date twice should produce the same result, not double-counted rows. Airflow doesn't enforce idempotency for you; it's a contract the pipeline author has to honour, typically by having each task overwrite a specific partition (`INSERT OVERWRITE PARTITION` or equivalent) rather than blindly appending. Teams that skip this discipline discover the cost the first time a transient failure forces a retry and their revenue table quietly doubles.

Operators, sensors, and the task abstraction

A DAG's nodes are built from operators, which are just parameterised units of work — a BashOperator runs a shell command, a PythonOperator runs a Python callable, a `SnowflakeOperator` runs a SQL statement against a warehouse, and so on. This is deliberately generic: Airflow doesn't care what a task does, only when it should run relative to its neighbours and what its retry policy is. Each task can independently specify a number of retries, a delay between them (often with exponential backoff), a timeout, and alerting hooks, so a flaky API call that fails one time in twenty can be retried automatically without failing the whole pipeline or waking anyone up.

Sensors are a special kind of operator that don't do work so much as wait for a precondition — a FileSensor polls until a file appears in cloud storage, an ExternalTaskSensor waits for a task in a different DAG to finish, and a partition sensor waits until a specific date's data has landed in a source table. Sensors let you express cross-system dependencies ('don't start this transform until the ingestion job three teams over has written today's data') without hard-coding a fixed wait time, which is the same category of bug cron encouraged in the first place. Modern Airflow versions also support 'deferrable' sensors that release their worker slot while waiting, so a fleet of pipelines waiting on slow upstream systems doesn't starve the scheduler of capacity.

The scheduler loop and what actually runs where

Under the hood, the Airflow scheduler continuously parses your DAG definition files (Python scripts that build the graph objects), evaluates which task instances are eligible to run based on their dependencies and the DAG's schedule, and pushes eligible tasks onto a queue. A separate executor then decides how those tasks are actually run: the simplest, SequentialExecutor, runs one task at a time and is only fit for local testing; the CeleryExecutor and KubernetesExecutor distribute tasks across a pool of workers or spin up a fresh pod per task, respectively, which is what production deployments actually use. This separation — scheduler decides what should run, executor decides where — mirrors the same divide-and-conquer principle seen in YARN's ResourceManager versus NodeManager split.

Visually, a rendered DAG in the Airflow UI looks like a factory floor: boxes for each task, coloured by state (queued, running, success, failed, upstream_failed), connected by arrows showing the flow of dependency, with a Gantt-style view showing exactly how long each task took relative to its neighbours. This visualisation is not incidental — it's the primary debugging tool, because when something breaks at 3am, the first question is always 'which node in the graph turned red, and did its failure cascade downstream,' and the DAG view answers both at a glance.

Frequently Asked Questions

What does 'DAG' actually mean in Airflow?

It stands for Directed Acyclic Graph: a set of tasks connected by directional dependency edges, with no cycles allowed, which guarantees there's always a valid order in which the tasks can execute.

Why is idempotency so important for Airflow tasks?

Because Airflow retries failed tasks and supports re-running historical dates for backfills, a task that isn't idempotent — for example one that appends rather than overwrites a partition — will silently produce duplicated or incorrect data on any retry or rerun.

What's the difference between the scheduler and the executor?

The scheduler parses DAGs and decides which task instances are ready to run based on dependencies and schedule; the executor is a separate component that decides how and where those ready tasks are actually executed, whether locally, via a Celery worker pool, or as Kubernetes pods.

How is a backfill different from a normal scheduled run?

A backfill triggers DAG runs for a range of past logical execution dates rather than waiting for the schedule to reach them, letting you reprocess historical data with an updated pipeline without waiting real days for the scheduler to catch up.