🧪 Data Validation in ETL Pipelines

About this simulation

Before raw sensor or log data ever reaches a machine-learning model, it usually passes through a validation gate. One of the simplest and most effective checks is the IQR (interquartile range) fence: compute the 25th and 75th percentiles (Q1, Q3) of recent values, then flag anything outside [Q1 − k·IQR, Q3 + k·IQR] as a likely sensor glitch, transmission error, or genuine anomaly worth a second look.

How it works

• Each glowing sphere is one incoming record; its height on the track encodes its sensor value.
• The translucent green slab is the current "valid" band — everything between the lower and upper fence.
• Records are colored indigo while in flight, then flash green (passed) or red (rejected) as they cross the gate.
• Rejected records are diverted down into the quarantine bin instead of reaching the model at the far end.

Controls

• IQR fence (k) — how many interquartile ranges beyond Q1/Q3 count as acceptable (1.5 is the classic default; lower is stricter).
• Outlier injection rate — how often a simulated sensor glitch or extreme reading appears in the stream.
• Flow speed — how fast records travel down the pipeline.
• ETL mode — streaming recomputes the fence continuously from a rolling window; batch recomputes it once per arriving batch, trading responsiveness for stability.

Did you know?

Batch ETL jobs can catch a systemic sensor drift by comparing whole batches against history, but they add latency; streaming pipelines validate record-by-record with low latency but can be fooled by a short burst of correlated bad data — which is why many production systems run both.