DexPoint: Generalizable Point Cloud Reinforcement Learning for
Sim-to-Real Dexterous Manipulation

Yuzhe Qin*1, Binghao Huang*1, Zhao-Heng Yin2, Hao Su1, Xiaolong Wang1
1UC San Diego, 2HKUST

*Equal Contribution
Conference on Robot Learning (CoRL) 2022



We propose a sim-to-real framework for dexterous manipulation which can generalize to new objects of the same category in the real world. The key of our framework is to train the manipulation policy with point cloud inputs and dexterous hands. We propose two new techniques to enable joint learning on multiple objects and sim-to-real generalization: (i) using imagined hand point clouds as augmented inputs; and (ii) designing novel contact-based rewards. We empirically evaluate our method using an Allegro Hand to grasp novel objects in both simulation and real world. To the best of our knowledge, this is the first policy learning-based framework that achieves such generalization results with dexterous hands.

Point Cloud Policy

We train dexterous manipulation using RL with single-viewed point cloud. The blue points are observed by camera
while the red points are imagined from robot model.

Grasp Different Objects

Open Door with Different Handles

Sim&Real Comparison

The point cloud in simulation and in real world are similar, which is a key factor for successful Sim2Real transfer.

Point Cloud in Real World
Point Cloud in Simulation

Failure Cases

We visualize the failure cases of our method on real robot. The video is 2x speed up.

(i) Unstable grasp due to insufficient contact

(ii) Bad finger motion after grasp


  title          = {DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulation },
  author         = {Qin, Yuzhe and Huang, Binghao and Yin, Zhao-Heng and Su, Hao and Wang, Xiaolong},
  journal        = {Conference on Robot Learning (CoRL)},
  year           = {2022},