⚡ Apache Spark Distributed Execution

The glowing node at the top is the driver — the single process that builds the physical plan, splits each stage into tasks, and coordinates the run. The ring of nodes below are executors, each an independent JVM process on a worker machine that runs tasks in parallel and reports back to the driver.

With shuffle on, every stage boundary is followed by a synchronisation phase where data particles fly between every pair of executors before the next stage can start, exactly like a groupBy or join that isn't already co-partitioned. With shuffle off, stages pipeline back-to-back with no cross-executor traffic, showing why narrow transformations are so much cheaper than wide ones.

Amdahl's law: only the parallel portion of the work (per-partition task execution) speeds up with more executors — the serial portion (driver planning and, when shuffle is on, the synchronisation barrier) does not. As executors increase, the theoretical speedup curve flattens toward 1/(1−p).

It sets how many tasks (partitions) each executor processes concurrently in a stage, similar to spark.executor.cores. Higher parallelism launches more particles per executor per stage, raising throughput until the visualisation's fixed executor slots become the bottleneck.