⚡ Inside Apache Spark's Distributed Execution Model

About this simulation

A Spark application has one driver that builds a logical plan (a DAG of RDD/DataFrame transformations) and a set of executors that actually hold data partitions in memory and run tasks. Transformations like map or filter are lazy — Spark just records them. Nothing runs on the cluster until an action like collect() or count() forces execution.

How it works

• Stage 1 — narrow transform: each partition is processed independently, in place, on the executor that already holds it. No network traffic.
• Shuffle: a wide transformation (like groupByKey or a join) needs matching keys together, so partitions are hashed and redistributed across executors over the network — the most expensive step in a Spark job.
• Stage 2 — reduce: each executor aggregates the partitions it now holds into a smaller result.
• Collect: final results are pulled back to the driver.

Controls

• Executors — how many worker processes make up the cluster.
• Partitions — how many chunks the input RDD/DataFrame is split into.
• Playback speed — animation speed multiplier.
• Cache RDD in memory — when on, Stage 1's output partitions stay resident (glowing) on their executors instead of being recomputed on the next run — the same benefit .cache()/.persist() gives on iterative workloads.
• ▶ Run job — fires the action that triggers the whole lazy DAG. ↺ Reset — clears the cluster back to an unevaluated state.

Did you know?

Because Spark keeps intermediate results in executor memory instead of writing every stage to disk (as classic MapReduce does), iterative algorithms — like gradient descent over many epochs — can be an order of magnitude faster, since only the first pass pays the cost of materializing the cached data.