Hadoop split "big data" processing into three cooperating systems. HDFS (Hadoop Distributed File System) breaks a large file into fixed-size blocks and replicates each block onto several DataNodes, while a single NameNode keeps the metadata — which blocks belong to which file, and where their replicas live. YARN (Yet Another Resource Negotiator) then schedules compute containers onto whichever nodes already hold the data, and MapReduce runs the actual job as three phases: map (process each block locally), shuffle (group intermediate results by key across the network) and reduce (aggregate each group into a final answer).
Hadoop's default HDFS block size grew from 64MB to 128MB precisely because network shuffle and NameNode metadata overhead both scale with block count — fewer, bigger blocks meant less coordination work per byte processed.
A 3D data-centre model of a Hadoop cluster: HDFS splits an input file into blocks and replicates them across DataNodes, then a MapReduce job runs with YARN scheduling map tasks onto the nodes that already hold the data before shuffling and reducing the results.
Why Hadoop separated storage (HDFS), scheduling (YARN) and computation (MapReduce) into three cooperating layers, and how "moving computation to the data" avoids shipping huge files across the network.
Adjust cluster size, replication factor and file blocks to reshape HDFS storage, then press Run MapReduce job and watch map, shuffle and reduce phases sweep across the cluster. Toggle the view to see YARN's container allocation instead of raw blocks.
Before YARN (Hadoop 2.0, 2013), MapReduce's JobTracker handled both resource scheduling and job logic — a bottleneck YARN removed by becoming a general-purpose cluster scheduler that today also runs Spark, Tez and other frameworks.