Compared to traditional data-driven learning methods, recently developed deep reinforcement learning (DRL) approaches can be employed to train robot agents to obtain control policies with appealing performance. However, learning control policies for real-world robots through DRL is costly and cumbersome. A promising alternative is to train policies in simulated environments and transfer the learned policies to real-world scenarios. Unfortunately, due to the reality gap between simulated and real-world environments, the policies learned in simulated environments often cannot be generalized well to the real world. Bridging the reality gap is still a challenging problem. In this paper, we propose a novel real–sim–real (RSR) transfer method that includes a real-to-sim training phase and a sim-to-real inference phase. In the real-to-sim training phase, a task-relevant simulated environment is constructed based on semantic information of the real-world scenario and coordinate transformation, and then a policy is trained with the DRL method in the built simulated environment. In the sim-to-real inference phase, the learned policy is directly applied to control the robot in real-world scenarios without any real-world data. Experimental results in two different robot control tasks show that the proposed RSR method can train skill policies with high generalization performance and significantly low training costs.

In this paper, a novel self-adaptive underactuated robot hand with rigid-flexible coupling fingers (SAU-RFC hand) is proposed. The seven degrees of freedom (DOFs) SAU-RFC hand is driven by four servomotors, consists of three fingers, including two side-turning (ST) fingers and one non-side-turning finger. Specially, the ST fingers can perform synchronous reverse rotation laterally with each other. Each finger with three joints and two DOFs introduces a flexible structure, and the inner part of the proximal phalanx that makes most of the contact with the object is replaced by a flexible belt. The fingers can generate flexion/extension under the pull of the flexible belt, and the middle and distal phalanxes are mechanically coupled through a four-bar linkage. In particular, the flexible belt in the inner direction of the finger will deform, while it will not deform in the outer direction since the outer is a rigid structure. The flexible belt not only plays the role of transmitting power but also has the effect of uniformizing the contact force. Due to the rigid-flexible finger structure, the developed robot hand has a higher self-adaptive grasping ability for objects with different shapes, sizes, and hardness. In addition, the kinematic and kinetic analyses of SAU-RFC hand are performed. A contact force distribution model is established for the flexible belt, which demonstrates its effect of promoting uniform force distribution theoretically. In the end, experiments are conducted on different objects to verify the performance of SAU-RFC hand.

Long-horizon dexterous tasks have been a long-standing problem in robotic manipulation. Previous studies have developed task and motion planning, imitation learning, and reinforcement learning methods for long-horizon manipulations. However, these methods are hard to achieve efficient planning for new tasks. Empowered with the Vision Language Model (VLM), recent studies significantly improve the generalization of robot systems. However, these works are only verified in simple pick-and-place tasks due to limited skills. To this end, we propose the Learn-Gen-Plan (LGP), which combines the VLM and learning-based primitives to endow robots with the ability to efficiently plan and complete various long-horizon dexterous tasks. LGP contains two key phases: skill generation and task planning. In skill generation, the Skill Generator is proposed to utilize the learned key primitives and hand-crafted trivial primitives to generate adaptive robot skills. In task planning, the Multimodal Planner generates the robot plan based on image observation, generated skills, and text prompts. We set up a series of dexterous tasks (e.g., cable routing, peg-in-hole assembly) in a real-world lighting circuit wiring scenario to evaluate LGP. The experimental results show that LGP efficiently generates robot plans with learned skills, controlling the robot to complete various multi-step cable wiring tasks.

In unstructured environments, robotic grasping tasks are frequently required to interactively search for and retrieve specific objects from a cluttered workspace under the condition that only partial information about the target is available, like images, text descriptions, 3D models, etc. It is a great challenge to correctly recognize the targets with limited information and learn synergies between different action primitives to grasp the targets from densely occluding objects efficiently. In this paper, we propose a novel human-like push-grasping method that could grasp unknown objects in clutter using only one target RGB-D image, called PLOT. First, we propose a target recognition method which automatically segments the objects both from the query image and workspace image, and extract the robust features of each segmented object. Through the designed feature matching criterion, the targets could be quickly located in the workspace. Second, we introduce a self-supervised targetoriented grasping system based on synergies between push and grasp actions. In this system, we propose a salient Q-learning framework that focuses the Q value learning in the area including targets and a coordination mechanism that selects the proper actions to search and isolate the targets from the surrounding objects, even in the condition of targets invisible. Our method is inspired by the working memory mechanism of human brain and can grasp any target object shown through the image and has good generality in application. Experimental results in simulation and real-world show that our method achieved the best performance compared with the baselines in finding the unknown target objects from the cluttered environment with only one demonstrated target RGB-D image and had the high efficiency of grasping under the synergies of push and grasp actions.

“Picking out the impurities” is a typical scenario in production line which is both time consuming and laborious. In this article, we propose a target-oriented robotic push-grasping system which is able to actively discover and pick the impurities in dense environments with the synergies between pushing and grasping actions. First, we propose an attention module, which includes target saliency detection and density-based occluded-region inference. Without the necessity of expensive labeling of semantic segmentation, our attention module can quickly locate the targets in the view or predict the candidate regions where the targets are most likely to be occluded. Second, we propose a push–grasp synergy framework to sequentially select proper actions in different situations until all targets are picked out. Moreover, we introduce an active pushing mechanism based on a novel metric, namely Target-Centric Dispersion Degree (TCDD) for better grasping. TCDD describes whether the targets are isolated from the surrounding objects. With this metric, the robot becomes more focused on the actions around the targets and push irrelevant objects away. Experimental results on both simulated environment and real-world environment show that our proposed system outperforms several baseline approaches,which also has the capability to be generalized to new scenarios.