Inside Apache Spark's Distributed Execution Model

How Spark's driver-executor split, lazy evaluation and in-memory shuffle let it outperform MapReduce on iterative workloads, and what actually happens when a Spark job runs.

▶ Open the simulation

Why MapReduce wasn't enough

MapReduce's discipline of writing every intermediate result to disk between stages was safe but slow, and the cost showed up most painfully in iterative algorithms — training a logistic regression model via gradient descent, say, which might need to sweep over the same dataset fifty or a hundred times. Under MapReduce, each of those fifty passes was a fresh job: read from HDFS, compute, write back to HDFS, repeat. Spark's founding bet, developed at UC Berkeley's AMPLab, was that if a cluster has enough aggregate RAM to hold the working dataset — which for many real workloads it does — then keeping data resident in memory across iterations rather than round-tripping through disk each time could turn a job that took hours into one that took minutes.

The abstraction that made this safe to do at scale was the Resilient Distributed Dataset, or RDD: a collection partitioned across the cluster that Spark can reconstruct after a node failure not by replicating the data itself, but by replaying the sequence of transformations that produced it, using a recorded lineage graph. This is a meaningfully different trade-off from HDFS's block replication — instead of paying storage cost upfront for redundancy, Spark pays a recomputation cost only if and when a failure actually happens, which is rare enough that the amortised cost is lower.

Lazy evaluation and the query plan

Spark code reads as if it executes eagerly — you call `.filter()`, then `.groupBy()`, then `.agg()` — but none of those calls actually touch data. Each one just appends a node to a logical plan, a directed graph of transformations, and nothing runs until an action is called, like `.collect()`, `.write()`, or `.count()`. This laziness isn't merely an implementation detail; it's what allows Spark's Catalyst optimiser to see the entire chain of operations at once and rewrite it before execution — pushing a filter down so it runs before a join rather than after, pruning columns nobody asked for, or reordering joins to put the smallest table first.

A naive alternative — executing each transformation immediately, one at a time — would waste enormous amounts of work: filtering rows only to discard most of a joined table's columns two steps later, when the filter could have run first and shrunk everything downstream. Because Catalyst sees the whole plan up front, it can apply exactly this kind of predicate and projection pushdown automatically, without the developer ever hand-tuning the operation order — turning what would be a sequence of naive full-dataset passes into a plan that touches the minimum data necessary at each stage.

The driver, the executors, and the shuffle

A running Spark application has one driver process, which holds the SparkContext, builds the logical and physical plans, and coordinates everything, and a set of executor processes spread across the cluster's worker nodes, each running in its own JVM with a slice of CPU cores and memory. The driver breaks the physical plan into stages, splits each stage into many small tasks — one task per data partition — and ships those tasks to executors to run in parallel. Executors report results and heartbeats back to the driver, which is also the single point of failure for the whole application: if the driver dies, the job dies, even if every executor is healthy, which is why production deployments run the driver on a resilient node and, for long-running or critical jobs, may checkpoint state so a restart doesn't have to redo everything.

Stage boundaries are determined by shuffles — any operation, like a `groupBy` or a `join` on non-co-partitioned data, that requires records to move between partitions across the network. Within a stage, tasks run independently and never need to talk to each other, which is why Spark can pipeline multiple narrow transformations (map, filter) into a single stage without materialising intermediate results. A shuffle, by contrast, forces a hard synchronisation point: every task in the upstream stage must finish writing its shuffle output before any task in the downstream stage can start reading it, because a downstream partition may need data contributed by every upstream task. This is the same conceptual bottleneck as MapReduce's shuffle, but Spark mitigates its cost by keeping shuffle outputs in memory or on local disk within the cluster rather than round-tripping through a distributed filesystem, and by letting the driver's plan minimise the number of shuffles a query needs in the first place.

DataFrames, partitions, and skew

Modern Spark code mostly uses the DataFrame API rather than raw RDDs, which adds a schema — column names and types — on top of the same partitioned, lazily-evaluated execution model, and this schema is precisely what lets Catalyst reason about columns and types well enough to do its optimisations. Under the hood, a DataFrame is still a collection of partitions spread across executors, and the number and size of those partitions materially affects performance: too few partitions and you leave cluster cores idle; too many and the scheduling overhead of managing thousands of tiny tasks starts to dominate the actual work.

The more insidious problem is data skew: if one join key — say, a single very popular product ID — accounts for a disproportionate share of the rows, the partition holding that key becomes enormous while its neighbours sit nearly empty, and the whole stage's wall-clock time collapses to whatever that one overloaded task takes, no matter how many idle executors are standing by. Spark's adaptive query execution can detect this at runtime and automatically split the oversized partition into several smaller ones, but engineers still routinely hand-tune around known skewed keys by salting them — appending a random suffix to spread a hot key across multiple partitions — because no automatic optimiser catches every real-world skew pattern.

Frequently Asked Questions

What is an RDD and do people still use it directly?

An RDD (Resilient Distributed Dataset) is Spark's original partitioned, fault-tolerant collection abstraction; most modern code uses the higher-level, schema-aware DataFrame API instead, but DataFrames still compile down to RDD operations under the hood.

Why is lazy evaluation faster than running each step immediately?

Because nothing executes until an action is called, Spark's optimiser can see the entire chain of transformations at once and rewrite it — pushing filters earlier, dropping unused columns, reordering joins — rather than executing each step naively in the order it was written.

What triggers a shuffle in Spark, and why is it expensive?

Operations like groupBy or joins on data that isn't already partitioned by the join key force records to move across the network between executors, which requires every upstream task to finish before downstream tasks can start, creating a hard synchronisation point that stalls parallelism.

What is data skew and why does it hurt Spark jobs?

Data skew happens when certain key values are far more common than others, causing some partitions to hold vastly more data than others; since a stage can't finish until its slowest task does, one oversized partition can dominate the total runtime even with many idle executors elsewhere.