Perception for Dexterous Manipulation

Sensing

High-resolution tactile and force sensors provide detailed information about contact forces and surface properties.

3D vision and state estimation techniques, such as stereo cameras and pose graph optimization, create a comprehensive understanding of the object's geometry and position in space.

Sensor calibration and synchronization are crucial for accurate data fusion, ensuring that measurements from different sensors align consistently across time and space.

Fusion and Policies

Learning-based fusion algorithms combine sensory information to make informed decisions about manipulation actions.

Contact-rich control strategies leverage tactile feedback to precisely guide the robot's end-effector during interactions with objects.

Robust failure recovery strategies are implemented to handle unexpected events, such as slips or collisions, ensuring continued operation.

Examples

Example: In-Hand Object Reorientation involves using multimodal data to determine the object’s orientation and applying precise forces to rotate it.

Collect multimodal data including tactile, visual, and force sensor readings during manipulation tasks.

Train policy with tactile-vision fusion techniques to learn optimal control strategies for complex manipulations.

Evaluate robustness on novel objects by systematically varying object properties and environmental conditions.

Frequently asked questions

Try it live

Everything above runs in your browser — open Inverse Kinematics (FABRIK) and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.