Grasp Multiple Objects with One Hand

Yuyang Li1,2,3, Bo Liu3, Yiran Geng4, Puhao Li1,2, Yaodong Yang1,3, Yixin Zhu3, Tengyu Liu1,†, Siyuan Huang1,†,
1National Key Laboratory of General Artificial Intelligence, BIGAI
2Dept. of Automation, THU 3Institute for AI, PKU 4School of EECS, PKU
†Corresponding Author
Paper arXiv Code Dataset (Coming Soon)
Teaser

Shadow Hand driven by our framework catches multiple Pokémons simultaneously.

The human hand's complex kinematics allow for simultaneous grasping and manipulation of multiple objects, essential for tasks like object transfer and in-hand manipulation. Despite its importance, robotic multi-object grasping remains underexplored and presents challenges in kinematics, dynamics, and object configurations. This paper introduces MultiGrasp, a two-stage method for multi-object grasping on a tabletop with a multi-finger dexterous hand. It involves (i) generating pre-grasp proposals and (ii) executing the grasp and lifting the objects. Experimental results primarily focus on dual-object grasping and report a 44.13% success rate, showcasing adaptability to unseen object configurations and imprecise grasps. The framework also demonstrates the capability to grasp more than two objects, albeit at a reduced inference speed.

Pipeline of MultiGrasp

Pipeline

We generate a pre-grasp pose with either the augmented Differentiable Force Closure (DFC) (A) or a diffusion model (B), then execute the grasp with a planned reaching trajectory (C) and learned lifting policy (D). We further distill the policy to a vision-based student for real-world scenarios (E).

Grasp 'Em: A Large-Scale Multi-Object Grasping Dataset

MultiGrasp released a large-scale dataset of multi-object grasping with a Shadow Hand that includes approximately 90k grasps for 8 different objects (73.7k double-object grasps, and 16.4k single-object grasps).

bulb+bulb bulb+camera bulb+cube knob+knob knob+duck knob+pear
camera+bulb camera+camera camera+cube duck+knob duck+duck duck+pear
cube+bulb cube+camera cube+cube pear+knob pear+duck pear+pear

Executing Multi-Object Grasps with a Two-Stage Policy

MultiGrasp executed with an optimization-based motion plan to guide the hand to the pre-grasp pose, followed by an RL policy for lifting the objects.

bulb+camera camera+knob cube+cylinder
duck+knob cylinder+duck bulb+bulb

Scalability to More Objects

MultiGrasp can generalize to grasping more objects. We showcase the execution trajectories of multiple cylinders, with their amount ranging from 3 to 5.

3 Objects 4 Objects 5 Objects

Real Robot Experiments

MultiGrasp generates grasp trajectories that are plausible to execute on a real Shadow Hand robot.


Real Robot Grasp Execution with a Real Shadow Hand Robot