Before the fingers touch anything, the only pose estimate comes from the overhead camera — noisy, and briefly cut off ("occluded") whenever the gripper's own body passes between camera and object on the way down. The moment a fingertip makes contact, a second, far more precise but purely local tactile estimate becomes available. Inverse-variance fusion blends the two by trusting whichever sensor is currently more certain:
fused = (vision/σv² + tactile/σt²) / (1/σv² + 1/σt²)
weight(sensor) ∝ 1/σ² — noisier sensor counts for less
- Vision noise σ — how jittery each camera pose reading is; higher values make the vision-only marker wander further from the true object.
- Tactile noise σ — precision of the contact-point estimate once a finger touches the surface; always tighter than vision, but only exists after contact.
- Camera calibration bias — a constant offset between the camera frame and the gripper frame, as if the two were never aligned. Vision-only grasps inherit this bias directly; "Calibrate camera" zeroes it, the way a calibration fixture would in the real system.
- Occlusion — while the gripper body blocks the camera's view near the object, the vision estimate freezes and its effective uncertainty grows the longer it stays stale, so fusion automatically leans harder on tactile once contact is made.
- Fusion mode — switch to Vision only or Tactile only to see why relying on a single modality fails: vision alone carries bias and occlusion gaps forever; tactile alone has nothing to steer by until contact already happened, so it can miss the object entirely.
Real-world relevance: this is the core sensing pattern behind in-hand and pick-and-place manipulation — cameras get you close, tactile sensing gets you precise, and confidence-weighted fusion is what lets a controller trust each source exactly as much as it currently deserves.