Home▸AI & Machine Learning▸Data Validation in ETL Pipelines

🧪 Data Validation in ETL Pipelines

Watch sensor records flow through an IQR-based validation gate: tune the outlier fence, injection rate and batch vs streaming recalibration to see which records reach the model.

AI & Machine Learning3DAdvanced60 FPS

A stream of sensor records flows down a track toward an IQR-based validation gate; the gate's fence is set from recent data statistics, and anything outside it gets diverted into a quarantine bin instead of reaching the model.

🔬 What It Demonstrates

Each record's height encodes its sensor value. The green slab marks the current valid band — [Q1 − k·IQR, Q3 + k·IQR] — recomputed either continuously (streaming) or once per batch, showing the real trade-off ETL engineers face.

🎮 How to Use

Adjust the IQR fence multiplier and outlier injection rate, switch between streaming and batch recalibration, and watch how the fence position and rejection rate respond in real time.

💡 Did You Know?

The IQR fence (typically k = 1.5) is one of the oldest robust outlier rules, dating to John Tukey's exploratory data analysis work — and it is still a first line of defence in modern ML feature pipelines.

⚙ Under the hood

Watch sensor records flow through an IQR-based validation gate: tune the outlier fence, injection rate and batch vs streaming recalibration to see which records reach the model.

machine learningdata validationetl pipelinesoutlier detectionstatistical analysisalgorithmsThree.js

3D · Three.js / WebGL renderer · 60 FPS target · runs fully client-side, no install