Handheld AR (ARKit / ARCore style) has no GPS-grade position sensor for centimetre accuracy indoors, so it estimates the camera's 6-DoF pose purely from what the camera sees — this is visual(-inertial) SLAM: Simultaneous Localization And Mapping.
Each frame:
1. Detect distinctive image features (corners, texture)
2. Match them to features seen in previous frames
3. Solve for the camera pose that best explains
how those matched points moved (triangulation)
4. Cluster co-planar features → detected surface (plane)
This simulator makes the pipeline visible and lets you break it:
- Feature points (cyan dots) are the trackable visual detail the algorithm relies on. Orbit the view — points currently inside the camera frustum turn bright (tracked); points outside go dim.
- Tracking noise stands in for real failure modes — motion blur, low light, a blank textureless wall. The fewer points are tracked and the higher the noise, the more the pose estimate drifts frame to frame.
- Anchor drift is the gap between the anchor's true position (dashed ring, fixed) and where the app actually renders the virtual object given its noisy pose estimate (solid model). Real AR content visibly "swims" relative to the world under exactly this condition.
- Detected plane grows as more co-planar feature points accumulate evidence for a flat surface — this is why AR apps ask you to slowly pan across a table before letting you place an object.
- Place anchor here re-anchors the model to the plane at the camera's current look-at point and zeroes drift, like tapping to place an object in a real AR app.
- Relocalize mimics loop closure: the tracker recognises a previously-mapped area and snaps the drift back to zero.
Confidence formula used here: confidence = trackedRatio × (1 − 0.6 × noise), and drift grows at a rate proportional to noise × (1 − trackedRatio) each frame while decaying proportionally to confidence — a simplified but directionally accurate model of real visual-inertial odometry error.