Mixed reality registration (3D)

Why a virtual overlay never sits exactly on its real marker: pixel noise in the camera pose estimate and the delay between head motion and the picture.

Parameters

Camera

Quality

Model checks

The science

What is modelled

A headset camera with focal length f (in pixels) tracks a flat square marker of side S at distance Z. The tracker finds the four marker corners with Gaussian pixel noise σ and turns them into a pose. The virtual overlay is then drawn from that pose, and the picture reaches the eye τ seconds after the head pose was measured.

Equations

  • Pinhole projection: a point at lateral offset X and depth Z lands at u = f X / Z pixels. One pixel therefore stands for Z/f metres at the marker.
  • Apparent side length s = f S / Z pixels, so depth is recovered as Ẑ = f S / s̄ where s̄ is the mean of the four measured sides.
  • Error propagation: each side length is a difference of two noisy corners (std σ√2); the four sides share corners, and the mean has variance σ²/2. Hence σZ = Z² σ / (√2 f S). Depth error grows with the square of distance.
  • Centre (lateral) error from the mean of four corners: σX = Z σ / (2 f).
  • Latency: with head yaw θ(t) = A sin(2π fh t) the overlay is drawn with the stale angle θ(t - τ). The angular error is A[sin ωt - sin ω(t-τ)], whose amplitude is exactly 2A |sin(ωτ/2)| ≈ ω A τ. At the marker it is displaced by about Z times that angle.

Checks run in this page

  • Monte Carlo depth/centre noise: 20,000 random corner sets are turned into Ẑ and compared with the formulas above (within 4 %, valid while σ is small compared with the side s in pixels).
  • Latency amplitude: the delayed sine difference is scanned numerically over five periods and compared with 2A|sin(ωτ/2)|.
  • Pixel scale: a point is projected through a real Three.js perspective camera with the same focal length, and the result must equal f X / Z pixels (independent of the formula used above).

Honest simplifications

  • The marker is seen front-on and only the size-based depth estimate is analysed. Real PnP solvers also have a strong tilt/depth coupling for oblique markers, which is not modelled.
  • Corner noise is white Gaussian and independent; real detectors have correlated, lighting-dependent errors, and cameras have distortion that is ignored here.
  • Latency is a pure delay of the head angle with a prescribed sinusoidal motion. Real devices partly cancel it with prediction and late-stage reprojection, which is not simulated.
  • The orange offsets in the 3D view are multiplied by the "visual exaggeration" slider so that millimetres are visible; the numbers in the read-outs are the true values.

The headset, marker board and table are Blender-built meshes with a baked wood texture; the tracking noise and the overlay are computed live in your browser.

Loading model...
Loading Blender model...
Latency lag of the overlay (mm, true scale)
Pose noise against distance (mm)
Peak latency error against delay (mm)
Made with 🟠 Blender · Three.js