This paper presents a comprehensive survey of deep learning approaches for dexterous robotic grasping, emphasizing recent progress enabled by multi-modal models and data-driven techniques. These developments have enabled the generation and execution of stable, context-aware grasps that can be conditioned on natural language, generalize across robot embodiments, and perform effectively in real-world settings. We organize our survey into three parts: (1) Datasets, the foundation for data-driven approaches, covering large-scale efforts that support learning-based grasping; (2) Grasp Synthesis, including diverse representation methods, generative modeling, and optimization-based techniques; and (3) Grasp Execution, encompassing reinforcement learning, imitation learning, heuristic control, and hybrid frameworks that translate grasps into executable actions. We also examine existing benchmarks and metrics for evaluating grasp plausibility, stability, and task alignment. We identify persistent challenges that bottleneck progress and discuss promising future directions to guide researchers toward building more general-purpose, robust dexterous manipulation systems..
A summary of key dexterous grasping datasets, highlighting their source, scale, and unique features.
| Dataset (Year) | Type | Hand | #Objects | #Grasps/Data | Modalities | Eval. Annotations | Unique Features |
|---|---|---|---|---|---|---|---|
| DexGraspNet (2022) | Sim | ShadowHand | 5,355 | 1.32M grasps | None (poses only) | Force-closure (optimized) | Very large simulated dataset; validated via an Isaac Gym checker |
| DexGrasp Anything (2024) | Hybrid (Real+Sim) | ShadowHand (+ human) | 15,698 | 3.40M grasps | Meshes, 6-DOF poses | Force closure, penetration | Largest-to-date dexterous dataset; curated and physics-expanded |
| RealDex (2024) | Real | ShadowHand | 52 | ~59K grasps | RGB-D (multi-view) | Human annotation (quality) | Teleoperated dexterity; multi-view capture; human-like patterns |
| MultiDex (2023) | Sim | ShadowHand | 58 | 16K grasps | None | Optimized grasps | Small-scale high-quality set for dexterous pre-training |
| UniDexGrasp (2023) | Sim | ShadowHand | 5,519 | 1.12M grasps | None | Optimized grasps | Million-scale dataset; 200+ grasps per object |
| GRAB (2020) | Real | Human (MANO retarget) | 51 | 1.64M grasps | RGB-D (multi-view) | Human grasp labels | Real human grasps; learn hand–object interactions |
| HO3D (2021) | Real | Human (MANO) | 10 | 77K frames | RGB-D (hand+object) | 6D hand+object poses | Egocentric videos; synchronized 6D pose annotations |
| DexYCB (2021) | Real | Human (MANO) | 20 | 582K frames | RGB-D | 6D hand+object poses | Multi-camera capture; dense annotations |
Methods for generating grasp candidates from object and task information.
Grasp execution strategies fall into two main families: classical methods that rely on explicit models and rules, and learning-based methods that acquire skills from data. Click a button to compare them on the interactive chart.
Commonly‑used metrics for stability, accuracy, task‑alignment, and diversity in dexterous grasping.
| Category | Metric | Description (incl. formula) | Goal |
|---|---|---|---|
| Stability & Plausibility | Simulation Success Rate | % of grasps that stay stable in a physics engine under gravity & disturbances. | ↑ |
| Q1 / Epsilon Quality | Radius of the largest ball in the grasp‑wrench space: \[ Q_1 = \max\bigl\{r\;|\;\{w: \|w\|\le r\}\subset W\bigr\} \] | ↑ | |
| Penetration Vol./Dist. | Overlap between hand & object meshes. \[ PD(H,O)=\max_{x\in H\cap O} d(x),\quad V_{\text{int}}=\text{Vol}(H\cap O) \] | ↓ | |
| Dataset Accuracy | Chamfer Distance (CD) | Point‑cloud distance \[ \tfrac1n\sum_{x\in X}\!\min_{y\in Y}\|x-y\|^2 + \tfrac1m\sum_{y\in Y}\!\min_{x\in X}\|y-x\|^2 \] | ↓ |
| MPJPE / MRRPE | Pose error for MANO / human hands. \[ \text{MPJPE}=\tfrac1K\sum_{k}\| (\hat J^k-\hat J^r)-(J^k-J^r)\|_2 \] | ↓ | |
| Task Alignment | Fréchet Inception Dist. | Distributional gap between generated & reference grasps (images or 3‑D). | ↓ |
| LLM / VLM Score | Vision‑or‑language‑model rating of grasp ↔ prompt consistency. | ↑ | |
| Diversity | Std. Dev. of Joint Angles | Variation of generated grasps for a single input. | ↑ |
↑ higher is better | ↓ lower is better
Typical shaping terms used in reinforcement-learning or optimization-based grasp controllers. All rewards are maximized.
| Reward Type | Purpose | Example Formulation |
|---|---|---|
| Approach | Move hand toward demonstrated pre-grasp pose | \[ r_{\text{approach}} = -\bigl\|\hat{\mathbf p}_{\text{pre}}-\mathbf p_{\text{hand}}\bigr\|_2, \qquad \text{if } \|\cdot\| > 0.2 \] |
| Pre-grasp Imitation | Align hand pose with a demonstrated pre-grasp | \[ r_{\text{pre}} = - \bigl(\|\mathbf p-\hat{\mathbf p}\|_2 + \| \mathbf q-\hat{\mathbf q}\|_2\bigr) \] |
| Trajectory Following | Track a desired object trajectory after grasp | \[ r_{\text{obj}} = - \bigl\|\mathbf x_{\text{obj}} - \mathbf x_{\text{target}}\bigr\|_2, \qquad t \ge \lambda \] |
| Contact | Ensure multi-finger contact with the object | \[ r_{\text{contact}} = \mathbf 1[\text{thumb} + 2\ \text{fingers contact}] \] |
| Force | Encourage proper force distribution on graspable surfaces |
Weighted sum of contact forces on graspable vs. non-graspable regions |
| Lift | Promote upward motion of the grasped object | \[ r_{\text{lift}} = z_{\text{obj}} - z_{\text{goal}} \] |
| Anatomical & Reg. | Maintain joint limits and smooth motions | Penalties on joint-limit violations and high velocities |
| Success Bonus | Reward on final task completion | \[ r_{\text{success}} = \mathbf 1[\text{goal reached}] \times b \] |
@{ADD HERE}