An ingest stream cuts incoming records into blocks (the way HDFS cuts a file into fixed-size chunks). Each block is hash-partitioned to one primary node, then copied to R−1 further nodes for fault tolerance — the blue beam is the primary write, the amber beams are replicas. Each node behaves like a single queueing server, so the whole cluster's capacity and lag follow the classic M/M/1 result:
μ = N · c · e / R (cluster capacity, records/s)
ρ = λ / μ (utilization)
W = 1 / (μ − λ) (mean time a record waits + is processed, ρ < 1)
- Cluster nodes (N) — more machines linearly raise capacity μ, so the same ingestion rate λ produces a lower utilization ρ and shorter latency W.
- Ingestion rate (λ) — Volume/Velocity in the "3 V's" of Big Data. Push it past capacity μ and ρ crosses 1: the cluster saturates and latency diverges, visible as nodes flushing red.
- Replication factor (R) — HDFS's answer to node failure: each block is written R times on different racks. It divides useful cluster capacity by R, since the same nodes must also service replica writes.
- Engine — Hadoop MapReduce dispatches work in coarse, disk-backed batches (a visible scheduling pulse plus disk-flush overhead); Spark keeps data in memory and processes it as a continuous micro-batch stream, raising per-node capacity c and cutting fixed overhead.
Real-world relevance: this same queueing math — capacity vs. arrival rate, replication's tax on useful throughput, batch vs. streaming trade-offs — is exactly what capacity planning for a real Hadoop/Spark/Kafka cluster comes down to.