Data Pipeline Backpressure Simulator

Pipeline Inputs

Ingestion rate 400 ev/s

Micro-batch size 50 ev

Worker count 3

Buffer capacity 14 batches

stable

Live Readouts

Queue depth

0 / 14

Throughput

0 ev/s

Avg latency

0 ms

Dropped batches

Utilization ρ = λ/μ

Sim time

0.0 s

How it works

A streaming ETL pipeline groups incoming events into fixed-size micro-batches, holds them in a bounded in-memory buffer (a queue), and hands each batch to one of several parallel worker processes. This is the core mechanic behind tools like Kafka consumers, Spark Structured Streaming and cloud data-warehouse loaders.

arrival rate:     λ = ingestion rate (events/s)
service capacity: μ = workers × (batch size / worker time-per-batch)
utilization:      ρ = λ / μ
Little's Law:      L = λ · W   (avg items in system = arrival rate × avg wait time)

Each worker consumes one batch at a time from the front of the queue (FIFO) and takes batchSize / workerRate seconds to finish it. New batches form as soon as enough events have accumulated (ingestRate × dt ≥ batchSize) and are pushed onto the buffer — unless the buffer is already at capacity, in which case the batch is dropped and the drop counter increments. This is exactly what real systems call backpressure: when arrivals outpace service capacity for long enough, something has to either queue, throttle, or drop.

Ingestion rate — how fast raw events arrive (λ). Push it past the workers' total capacity to trigger backpressure.
Micro-batch size — events grouped per unit of work; bigger batches take longer per worker-cycle but reduce per-batch overhead.
Worker count — parallel consumers draining the queue; more workers raise service capacity μ.
Buffer capacity — the maximum queue depth before batches start getting dropped instead of queued.

When ρ < 1 the system is stable and the queue empties out; when ρ ≥ 1 the queue grows without bound until it hits capacity and starts shedding load — the same tradeoff every real data-pipeline operator tunes buffer sizes and autoscaling policies around.

Queued batch Worker processing Dropped (buffer full)

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.