From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperation

Yuzhe Qin, Hao Su*, Xiaolong Wang*
UC San Diego
*Equal advising

We first collect teleoperation demo using the Customized Hand in the simulation, then translate the demo to an Allegro Hand by retargeting. Finally, we train policy using the translated demo and deploy to real robot.


We propose to perform imitation learning for dexterous manipulation with multi-finger robot hand from human demonstrations. We introduce a novel single-camera teleoperation system to collect the 3D demonstrations efficiently with only an iPad and a computer. One key contribution of our system is that we construct a customized robot hand for each user in the physical simulator, which is a manipulator resembling the same kinematics structure and shape of the operator's hand. This not only avoids unstable human-robot hand retargeting during data collection, but also provides a more intuitive and personalized interface for different users. This allows for scalable and high quality data collection. Once the data is collected, the customized robot hand trajectories can be converted to different specified robot hands (models that are manufactured and commercialized) to generate training demonstrations. We experiment with multiple complex manipulation tasks, and show our approach achieves large improvement over baselines over multiple specified robot hands.


Single Camera Teleoperation

Construct Customized Hand

We estimate the hand MANO shape parameters of human operator and construct a customized hand to use.

Teleoperate Customized Hand

Then the operator will use this customized robot hand to collect demonstration on dexterous manipulation task.

More Teleoperation Visualization

Multi Hand Demonstration Translation

Translate from the teleoperation trajectory with the customized hand to demonstration trajectory with any other dexterous robot hand, e.g. Allegro Hand. We use hand pose retargeting to translate the demonstrations.

Customized Hand

Allegro Hand

Schunk Hand

Adroit Hand

Demo Augmented Reinforcement Learning

With translated demonstrations, we train manipulation policy using demo augmented RL using DAPG. The trained policy with our demonstration shows great robustness when deployed to the real world.


  author         = {From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperation},
  title          = {Qin, Yuzhe and Su, Hao and Wang, Xiaolong},
  archivePrefix  = {arXiv},
  year           = {2022},