⚡ Inside Apache Spark's Distributed Execution Model

One driver plans the job and splits each stage into tasks; a ring of executors runs those tasks in parallel. Tune the cluster below and watch tasks stream out, and — when a shuffle is due — watch every executor exchange data before the next stage can begin.

Cluster

Live metrics

Stage1 / 6
Active executors0
Tasks completed0
Throughput0 tasks/s
Amdahl speedup1.00×
driver idle executor running task shuffle
Speedup(N) = 1 / ((1−p) + p/N)
p = parallel fraction of the stage (task execution).
(1−p) = serial fraction (driver planning + shuffle barrier).
As N → ∞, Speedup → 1/(1−p) — the Amdahl's-law ceiling.

How Spark's Execution Model Works

A running Spark application has exactly one driver — it builds the logical and physical plan, breaks it into stages at every shuffle boundary, and splits each stage into one task per partition. Tasks are shipped to executors, independent JVM processes spread across the cluster, which run them in parallel and report back. Within a stage tasks never talk to each other, so Spark pipelines narrow transformations (map, filter) freely — but a shuffle (groupBy, join on non-co-partitioned data) forces every task in the upstream stage to finish before any downstream task can start, a hard synchronisation point. Because only the per-partition task work parallelises across executors while planning and shuffle barriers stay serial, the achievable speedup follows Amdahl's law: Speedup(N) = 1 / ((1−p) + p/N), where p is the parallel fraction — which is exactly why adding executors helps a lot at first and then flattens out, especially once shuffles dominate the job.

About the Apache Spark Distributed Execution Simulator

Apache Spark replaced MapReduce's disk-round-trip discipline with an in-memory, lazily-evaluated execution model built around Resilient Distributed Datasets (RDDs). This simulation visualises the resulting driver/executor architecture: a single coordinating driver process plans the job as a directed graph of stages, and a pool of executor processes runs the resulting tasks in parallel across the cluster.

The most consequential design decision in that model is where synchronisation happens. Narrow transformations like map and filter can be pipelined within a stage with zero cross-executor communication, because each task only ever touches its own partition. A shuffle — triggered by a groupBy or a join on data that isn't already co-partitioned — is different: every upstream task must finish writing its output before any downstream task can start reading, because a downstream partition may depend on data from every upstream task. Toggle the shuffle control to see that hard barrier appear (or disappear) between stages.

The Executors and Task parallelism sliders let you explore the diminishing returns predicted by Amdahl's law: the live "Amdahl speedup" readout uses Speedup(N) = 1/((1−p)+p/N) with a parallel fraction p that shrinks whenever shuffle is enabled (because the shuffle barrier adds serial synchronisation time), matching why real Spark jobs dominated by shuffles scale worse with extra executors than shuffle-light jobs do.

Frequently Asked Questions

What do the driver and executors represent in this simulation?

The glowing node at the top is the driver — the single process that builds the physical plan, splits each stage into tasks, and coordinates the run. The ring of nodes below are executors, each an independent JVM process on a worker machine that runs tasks in parallel and reports back to the driver.

What does the shuffle toggle actually change?

With shuffle on, every stage boundary is followed by a synchronisation phase where data particles fly between every pair of executors before the next stage can start, exactly like a groupBy or join that isn't already co-partitioned. With shuffle off, stages pipeline back-to-back with no cross-executor traffic, showing why narrow transformations are so much cheaper than wide ones.

Why does adding executors eventually stop helping throughput?

Amdahl's law: only the parallel portion of the work (per-partition task execution) speeds up with more executors — the serial portion (driver planning and, when shuffle is on, the synchronisation barrier) does not. As executors increase, the theoretical speedup curve flattens toward 1/(1−p).

What is task parallelism controlling here?

It sets how many tasks (partitions) each executor processes concurrently in a stage, similar to spark.executor.cores. Higher parallelism launches more particles per executor per stage, raising throughput until the visualisation's fixed executor slots become the bottleneck.