Vision-Tactile Fusion for Grasping in 2D: Side-View Sensor Blend

Fusion Mode

Sensor Noise

Vision noise σ 0.35

Tactile noise σ 0.06

Camera calibration bias 0.20

Live Estimate Error

Vision error

–

Tactile error

–

Fused error

–

Success rate

–

Bottom strip: live target-error trace during descent. Bars: last 20 trial errors (dashed line = success threshold).

How it works

This is a side-on cross-section of the same gripper: before any finger touches the object, the only pose reading comes from the overhead camera — noisy, and briefly cut off ("occluded") whenever the gripper's own body passes between camera and object on the way down. The instant a fingertip makes contact, a second, far more precise but purely local tactile estimate appears. Inverse-variance fusion blends the two by trusting whichever sensor is currently more certain:

fused = (vision/σv² + tactile/σt²) / (1/σv² + 1/σt²)
weight(sensor) ∝ 1/σ²  — noisier sensor counts for less

Vision noise σ — how jittery the camera's horizontal-position reading is; higher values make the vision-only marker wander further from the true object.
Tactile noise σ — precision of the contact-point estimate once a finger touches the surface; always tighter than vision, but only exists after contact.
Camera calibration bias — a constant offset between the camera frame and the gripper frame. Vision-only grasps inherit this bias directly; "Calibrate camera" zeroes it.
Occlusion — while the gripper body blocks the camera's view near the object, the vision estimate freezes and its effective uncertainty grows the longer it stays stale, so fusion automatically leans harder on tactile once contact is made.
Fusion mode — switch to Vision only or Tactile only to see why relying on a single modality fails: vision alone carries bias and occlusion gaps forever; tactile alone has nothing to steer by until contact already happened, so it can miss the object entirely.
Error trace / trial bars — the bottom strip plots the live distance between the current target and the true object position while a trial runs; the bar history under it shows the final error of each of the last 20 trials against the success threshold (dashed).

Real-world relevance: this is the core sensing pattern behind in-hand and pick-and-place manipulation — cameras get you close, tactile sensing gets you precise, and confidence-weighted fusion is what lets a controller trust each source exactly as much as it currently deserves.

VISION OCCLUDED

True object position Vision estimate Tactile estimate Fused estimate (gripper target)

target error (live) + last 20 trials