🌬️ Apache Airflow and the DAG Model

About this simulation

Apache Airflow schedules data pipelines as directed acyclic graphs (DAGs) — a task only starts once every task it depends on has finished successfully. This scene shows one pipeline (extract → validate → transform → join → load → dbt models → notify) laid out left to right, with tasks changing color as the scheduler moves them through their lifecycle.

How it works

• Pending → Queued — a task becomes eligible the instant all of its upstream dependencies succeed, but it still has to wait for a free worker slot.
• Running — a queued task starts once the worker pool has capacity; a pulsing amber cube marks it in progress.
• Retry — tasks fail at the configured rate; Airflow automatically retries a failed task (up to 3 attempts) before marking the whole run failed.
• Success — a green cube fires a light pulse down every outgoing edge, unlocking downstream tasks — this is exactly how Airflow's scheduler reacts to task completion events.

Controls

• Run speed — how fast simulated task durations play out.
• Task failure rate — probability any single task attempt fails and needs a retry.
• Worker pool size — Airflow's parallelism/pool slots: how many tasks can run at once across all DAG runs.
• Backfill historical runs — queues several DAG runs back-to-back instead of one, the way Airflow's backfill command replays missed schedule intervals — watch them compete for the same worker pool.
• ↺ Restart scheduler — clears all runs and starts fresh. Drag to rotate, scroll to zoom.

Did you know?

Because Airflow models dependencies explicitly as a graph rather than as a chain of cron jobs, it can safely re-run just the failed branch of a pipeline, parallelize independent branches, and backfill months of missed runs — none of which a plain cron schedule can express.