The Hadoop Ecosystem: How HDFS, MapReduce and YARN Split the Work
A look at how Hadoop divided storage, processing and cluster scheduling into three cooperating systems, why that separation was revolutionary, and where it has since been overtaken.
Three problems, three subsystems
Before Hadoop, the standard answer to 'we have more data than one machine can hold' was to buy a bigger machine: more disks, more RAM, a fatter RAID array. That approach has a ceiling, and it is an expensive one. Hadoop's founding insight, borrowed from Google's internal GFS and MapReduce papers, was to stop scaling up and start scaling out — to spread both the data and the computation across hundreds or thousands of ordinary, failure-prone servers, and to make the software responsible for hiding those failures from the programmer.
To do that well, Hadoop split the problem into three distinct concerns that earlier systems had usually bundled together. The Hadoop Distributed File System (HDFS) is responsible only for durably storing bytes across a cluster. MapReduce is responsible only for expressing and executing a computation over that data. YARN — Yet Another Resource Negotiator — is responsible only for deciding which machines get to run which tasks at any given moment. Each layer can be reasoned about, and later replaced, independently of the other two, which is exactly what happened as the ecosystem matured.
HDFS: replication instead of RAID
HDFS takes every file you write and slices it into large blocks, typically 128 MB each — far bigger than a conventional filesystem's 4 KB blocks, because the goal is to minimise the overhead of seeking across spinning disks during sequential scans. Each block is then replicated, by default three times, and the copies are deliberately scattered: one replica on the writing node's rack, a second on a different node in the same rack, and a third on a node in a different rack entirely. A single NameNode holds the metadata — which blocks make up which file, and where each replica lives — entirely in memory for speed, while the actual bytes sit on DataNodes spread across the cluster.
This rack-aware placement is a deliberate trade-off between two failure modes. Keeping two replicas in the same rack means a block can be reconstructed quickly using only in-rack, high-bandwidth network links if one node dies. Keeping a third replica in a different rack protects against a whole rack going dark — a switch failure or a power distribution unit tripping, which happens more often in real datacentres than people expect. If a DataNode stops sending its periodic heartbeat, the NameNode marks its blocks under-replicated and schedules new copies elsewhere automatically, without an operator lifting a finger. The practical result is a filesystem that treats disk and server failure as a Tuesday, not an emergency.
MapReduce: computation that goes to the data
The genius of MapReduce was not the map/reduce abstraction itself — functional programmers had used map and fold for decades — but the decision to schedule each map task on, or very near, the node that already holds the relevant HDFS block. Moving a terabyte of data across a network to a compute node is slow; moving a few kilobytes of code to where the data already sits is nearly free. This principle, data locality, is why Hadoop clusters typically run storage and compute on the same physical machines rather than separating them, at least in the original design.
A MapReduce job runs in two strictly ordered phases connected by a costly middle step called the shuffle. The map phase reads input splits in parallel and emits key-value pairs — counting words, say, emits (word, 1) for every token. Between map and reduce, the framework partitions all emitted pairs by key, sorts each partition, and ships them across the network so that every value for a given key ends up on the same reducer; this shuffle-and-sort step is usually the most expensive part of the whole job, because it is the one phase that cannot be made local. The reduce phase then aggregates each key's values — summing the word counts — and writes the result back to HDFS. Every intermediate result between phases is written to disk, which is the single biggest reason MapReduce was eventually superseded for iterative workloads: a ten-step pipeline meant ten round trips to disk, even when the whole dataset would happily fit in the cluster's aggregate RAM.
YARN: separating scheduling from execution
In Hadoop's first generation, MapReduce's own JobTracker did double duty as both the resource manager for the whole cluster and the execution engine for MapReduce jobs specifically. That coupling meant the cluster could only ever run MapReduce; there was no way to also schedule a Spark job or an MPI job on the same machines without fighting over resources in an ad hoc way. YARN, introduced in Hadoop 2, pulled the resource-scheduling logic out into its own layer, general enough to host any kind of distributed application.
YARN's architecture mirrors a hiring manager delegating to project leads. A single ResourceManager tracks how much CPU and memory is free on every NodeManager (one per worker machine) and negotiates containers — bundles of a fixed CPU and memory allocation — on request. When a new job arrives, the ResourceManager first launches a small ApplicationMaster for that specific job inside one container; the ApplicationMaster then requests further containers to actually run the job's tasks, monitors their progress, and asks for replacements if any fail. This two-level design meant the cluster-wide scheduler no longer needed to understand the internals of MapReduce, Spark, Tez, or anything else — it only needed to hand out containers fairly, which is why a single YARN cluster could, from 2013 onward, run mixed workloads from several different processing engines simultaneously.
Where the ecosystem went from here
The three-way split proved its worth almost immediately: because MapReduce was decoupled from resource management, Apache Spark and Apache Tez could plug into YARN as alternative execution engines without anyone rewriting HDFS or the scheduler. Spark in particular kept intermediate data in memory across stages rather than round-tripping through disk after every step, which made it five to ten times faster on iterative machine-learning and graph workloads and effectively replaced MapReduce as the default processing engine within a few years, even on clusters that still used HDFS underneath.
HDFS itself has had a longer run but is now under pressure from a different direction: cloud object storage. Services like Amazon S3 offer similar durability guarantees, effectively unlimited scale, and — critically — decouple storage from any particular compute cluster, so you can spin compute up and down without ever moving the data. Many organisations that once ran on-premises Hadoop clusters have since migrated their data to S3 or equivalent object stores and kept only the processing-engine layer (Spark, Presto, Trino) running on top, which shows that the storage/compute/scheduling split Hadoop pioneered outlived Hadoop's own storage implementation. The lesson that stuck, even as individual components were swapped out, was architectural: separate concerns cleanly, and any one of them can be replaced as better options appear.
Frequently Asked Questions
Is Hadoop still used today?
Large legacy deployments still run HDFS and YARN, particularly in organisations with heavy on-premises investment, but new projects overwhelmingly favour cloud object storage plus Spark, or fully managed warehouses, because they avoid the operational burden of running a large cluster.
Why does HDFS use such large block sizes?
Large blocks (typically 128 MB) minimise the number of disk seeks needed to read through big files sequentially, which suits Hadoop's batch-analytics workloads far better than the small blocks used by general-purpose filesystems.
What exactly does YARN add that MapReduce alone didn't have?
YARN separates cluster-wide resource negotiation from the logic of any one processing framework, so multiple engines like Spark, Tez and MapReduce can share the same physical cluster and be scheduled fairly against each other.
Why was the shuffle phase such a bottleneck in MapReduce?
The shuffle requires sorting and transferring data across the network between every map and reduce stage, and because MapReduce writes intermediate results to disk at each stage boundary, multi-step pipelines paid that disk and network cost repeatedly.