Construct Customized Hand
We estimate the hand MANO shape parameters of human operator and construct a customized hand to use.
We first collect teleoperation demo using the Customized Hand in the simulation, then translate the demo to an Allegro Hand by retargeting. Finally, we train policy using the translated demo and deploy to real robot.
We propose to perform imitation learning for dexterous manipulation with multi-finger robot hand from human demonstrations. We introduce a novel single-camera teleoperation system to collect the 3D demonstrations efficiently with only an iPad and a computer. One key contribution of our system is that we construct a customized robot hand for each user in the physical simulator, which is a manipulator resembling the same kinematics structure and shape of the operator's hand. This not only avoids unstable human-robot hand retargeting during data collection, but also provides a more intuitive and personalized interface for different users. This allows for scalable and high quality data collection. Once the data is collected, the customized robot hand trajectories can be converted to different specified robot hands (models that are manufactured and commercialized) to generate training demonstrations. We experiment with multiple complex manipulation tasks, and show our approach achieves large improvement over baselines over multiple specified robot hands.
We estimate the hand MANO shape parameters of human operator and construct a customized hand to use.
Then the operator will use this customized robot hand to collect demonstration on dexterous manipulation task.
Translate from the teleoperation trajectory with the customized hand to demonstration trajectory with any other dexterous robot hand, e.g. Allegro Hand. We use hand pose retargeting to translate the demonstrations.
With translated demonstrations, we train manipulation policy using demo augmented RL using DAPG. The trained policy with our demonstration shows great robustness when deployed to the real world.
@misc{qin2022from,
author = {From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperation},
title = {Qin, Yuzhe and Su, Hao and Wang, Xiaolong},
archivePrefix = {arXiv},
year = {2022},
}