Question 1

What is sensor fusion in autonomous vehicle perception?

Accepted Answer

Sensor fusion combines measurements from multiple sensor types -- typically LIDAR, cameras, and radar -- into a single, more accurate estimate of the world than any sensor could produce alone. LIDAR returns precise 3D point clouds with reliable range but is sparse at long distance and struggles in fog or heavy rain. Cameras give dense, high-resolution images with excellent bearing (angular) accuracy and class information, but poor and noisy depth estimation from a single lens. A Kalman filter fuses these complementary strengths by weighting each measurement according to its own noise covariance, so the final track leans on whichever sensor is more trustworthy in that direction at that moment.

Question 2

How does a Kalman filter combine LIDAR and camera data?

Accepted Answer

Each tracked object keeps a state vector of position and velocity plus a covariance matrix describing the estimate's uncertainty. Every simulation step runs a predict step (advancing the state with a constant-velocity motion model and adding process noise) followed by one or more update steps, one per available measurement. Each update computes the innovation (the gap between the predicted position and the new measurement), scales it by the Kalman gain -- which is derived from the ratio of the current uncertainty to the sensor's own noise covariance -- and blends it into the state. A precise, low-noise LIDAR hit pulls the estimate strongly toward itself; a noisy camera depth reading pulls only weakly, while its accurate bearing component still helps a lot.

Question 3

Why do cameras and LIDAR have different noise characteristics in this simulation?

Accepted Answer

LIDAR measures range directly with a laser time-of-flight, so its error is small and nearly the same in every direction (isotropic), growing only slowly with distance. A single camera estimates 3D position by inference -- from bounding-box size, ground-plane geometry, or learned depth -- which is accurate for the angle to an object (bearing) but highly uncertain along the line of sight (depth). This simulation models that anisotropic behaviour explicitly: each camera detection's noise covariance is built as an ellipse stretched along the depth axis and rotated to match the object's true bearing from the ego vehicle, then correlated Gaussian noise is sampled from that ellipse using a Cholesky decomposition.

Question 4

What do the raw dots and the solid boxes represent?

Accepted Answer

The small translucent cyan spheres are raw, noisy LIDAR detections and the small translucent amber cubes are raw, noisy camera detections -- exactly what each sensor reports before any filtering. The solid orange boxes are the fused Kalman tracks: the filtered, smoothed state estimate for each obstacle after combining every available measurement with its correct covariance weighting. Watching the solid box stay steady while the raw dots jitter around it is the whole point of sensor fusion made visible.

Question 5

What happens when a sensor is switched off in this simulation?

Accepted Answer

Turning LIDAR or the camera off simply stops that sensor's detections from being generated for every subsequent frame. Existing tracks keep running on the predict step alone -- their position advances using the last known velocity and their uncertainty (covariance) grows every step because no new measurement is shrinking it. If a track goes too many frames without any update from either sensor it is deleted, which is exactly how real perception stacks handle sensor dropout or occlusion: coast briefly on the motion model, then give up if nothing confirms the object is still there.

Question 6

How does the tracker handle new obstacles appearing or disappearing?

Accepted Answer

Every frame, each new detection is compared against the predicted position of every existing track using a nearest-neighbour gate. Detections that fall within the gate update that track (a real object can be confirmed by both LIDAR and camera within the same frame). Detections that match nothing are clustered with any other unmatched detections nearby and spawned as a brand-new tentative track, which is promoted to a confirmed track (drawn as a fully solid box) only after a few consecutive hits. Tracks that stop receiving detections -- because an obstacle left the scene or a sensor was disabled -- accumulate misses and are deleted once they exceed the miss limit, so the active track count always reflects what the fused system currently believes is out there.

🚗 Waarneming bij Autonome Voertuigen — Sensorfusie Live

Over sensorfusie voor waarneming bij autonome voertuigen

Veelgestelde vragen

What did you find?