EN 
Українська
Español
Deutsch
Français🚗 Autonomous Vehicle Perception — Sensor Fusion Live
Fuse simulated LIDAR point clouds with camera-style detections using a Kalman-filter tracker, watching the fused object list stay locked onto moving obstacles in real time.
About Sensor Fusion for Autonomous Vehicle Perception
Autonomous vehicles never trust a single sensor. LIDAR delivers precise, nearly isotropic 3D range measurements but is sparse and struggles in bad weather; cameras are dense and cheap but only estimate depth indirectly, so their distance readings are noisy while their bearing (angle) readings are excellent. A Kalman filter is the classical, mathematically optimal way (for linear-Gaussian systems) to combine these complementary strengths: each tracked obstacle keeps a state estimate of position and velocity plus a covariance matrix describing how confident the tracker is, and every incoming measurement nudges that estimate by an amount proportional to how much more trustworthy the sensor is than the current estimate.
This simulation renders a genuine 3D scene around an ego vehicle: real Three.js meshes move along independent paths, a per-object Kalman filter predicts each obstacle's motion every frame with realistic process noise, and simulated LIDAR and camera measurements — each with their own, physically motivated noise covariance — are fused into a solid tracked bounding box. The raw, jittery sensor hits are drawn as small translucent markers so the smoothing benefit of fusion is directly visible rather than just claimed.
Frequently Asked Questions
What is sensor fusion in autonomous vehicle perception?
Sensor fusion combines measurements from multiple sensor types — typically LIDAR, cameras, and radar — into a single, more accurate estimate of the world than any sensor could produce alone. LIDAR returns precise 3D point clouds with reliable range but is sparse at long distance and struggles in fog or heavy rain. Cameras give dense, high-resolution images with excellent bearing accuracy and class information, but poor and noisy depth estimation from a single lens. A Kalman filter fuses these complementary strengths by weighting each measurement according to its own noise covariance, so the final track leans on whichever sensor is more trustworthy in that direction at that moment.
How does a Kalman filter combine LIDAR and camera data?
Each tracked object keeps a state vector of position and velocity plus a covariance matrix describing the estimate's uncertainty. Every simulation step runs a predict step (advancing the state with a constant-velocity motion model and adding process noise) followed by one or more update steps, one per available measurement. Each update computes the innovation — the gap between the predicted position and the new measurement — scales it by the Kalman gain, derived from the ratio of current uncertainty to the sensor's own noise covariance, and blends it into the state. A precise LIDAR hit pulls the estimate strongly toward itself; a noisy camera depth reading pulls only weakly, while its accurate bearing component still helps a lot.
Why do cameras and LIDAR have different noise characteristics in this simulation?
LIDAR measures range directly with a laser time-of-flight, so its error is small and nearly the same in every direction, growing only slowly with distance. A single camera estimates 3D position by inference, which is accurate for the angle to an object but highly uncertain along the line of sight. This simulation models that explicitly: each camera detection's noise covariance is built as an ellipse stretched along the depth axis and rotated to match the object's true bearing from the ego vehicle, then correlated Gaussian noise is sampled from that ellipse using a Cholesky decomposition.
What do the raw dots and the solid boxes represent?
The small translucent cyan spheres are raw, noisy LIDAR detections and the small translucent amber cubes are raw, noisy camera detections — exactly what each sensor reports before any filtering. The solid orange boxes are the fused Kalman tracks: the filtered, smoothed state estimate for each obstacle after combining every available measurement with its correct covariance weighting. Watching the solid box stay steady while the raw dots jitter around it is the whole point of sensor fusion made visible.
What happens when a sensor is switched off in this simulation?
Turning LIDAR or the camera off simply stops that sensor's detections from being generated for every subsequent frame. Existing tracks keep running on the predict step alone — their position advances using the last known velocity and their uncertainty grows every step because no new measurement is shrinking it. If a track goes too many frames without any update it is deleted, which is exactly how real perception stacks handle sensor dropout or occlusion: coast briefly on the motion model, then give up if nothing confirms the object is still there.
How does the tracker handle new obstacles appearing or disappearing?
Every frame, each new detection is compared against the predicted position of every existing track using a nearest-neighbour gate. Detections that fall within the gate update that track — a real object can be confirmed by both LIDAR and camera within the same frame. Detections that match nothing are clustered with any other unmatched detections nearby and spawned as a brand-new tentative track, promoted to a confirmed track only after a few consecutive hits. Tracks that stop receiving detections accumulate misses and are deleted once they exceed the miss limit, so the active track count always reflects what the fused system currently believes is out there.
Real per-object Kalman filters (constant-velocity state, process noise, covariance-weighted updates) fuse simulated LIDAR and camera detections in a genuine 3D Three.js scene, with nearest-neighbour data association handling track creation and deletion.
3D · Three.js / WebGL renderer · 60 FPS target · runs fully client-side, no install