🧪 Data Science Data Engineering Simulation

This simulation explores the practical aspects of data engineering by modelling a real streaming ETL pipeline: events arrive continuously, get grouped into micro-batches, wait in a bounded buffer, and are drained by a pool of parallel workers into a sink — exactly the pattern behind Kafka consumers, Spark Structured Streaming jobs and warehouse-loading pipelines. Tune the ingestion rate, batch size, worker count and buffer capacity to push the pipeline from comfortably stable into backpressure, and watch the buffer column fill in 3D while queue depth, throughput, average latency, dropped-batch count and utilization update live — a hands-on way to see why every data-engineering team spends so much time tuning buffer sizes, batch windows and autoscaling policies.