Assembly positioning by visual servoing (VS) is a basis for autonomous robotic assembly. In practice, VS control suffers potential stability and convergence problems due to image and physical constraints, e.g., field of view constraints, image local minima, obstacle collisions, and occlusion. Therefore, this article proposes a novel deep reinforcement learning-based hybrid visual servoing (DRL-HVS) controller for motion planning of VS tasks. DRL-HVS controller takes current observed image features and camera pose as inputs, and the core parameters of hybrid VS are dynamically optimized using a deep deterministic policy gradient (DDPG) algorithm to obtain an optimal motion scheme, considering image/physical constraints and robot motion performance. In addition, an adaptive exploration strategy is proposed to further improve the training efficiency by adaptively tuning the exploration noise parameters. In this way, the offline pretrained DRL-HVS controller in the virtual environment, where the DDPG actor–critic network is continuously optimized, can be quickly deployed to a real robot system for real-time control. Experiments based on an eye-in-hand VS system are conducted with a calibrated HIKVISION RGB camera mounted on the end-effector of a GSK-RB03A1 six degree-of-freedom (6-DoF) robot. Basic VS task experiments show that the proposed controller achieves better performance than the existing methods: the servoing time is 24% smaller than that of the five-dimensional VS method, a 100% success rate with the perturbed ranges of the initial position within 25 mm for translation and 20° for rotation, and a 48% efficiency improvement. Moreover, a planetary gear component assembly process case study, where the robot aims to automatically put the gears on the gear shafts, is conducted to demonstrate the applicability of the proposed method in practice.

For the large redundant manipulator, due to its long working distance and large mass, the number of links (i.e., manipulator’s arms) that can be driven to move simultaneously is limited. Otherwise, the control accuracy and motion stability of the manipulator will deteriorate. Focusing on that, a weighted Newton iteration (WNI) algorithm for trajectory planning of the manipulator is firstly proposed, where the motion of the manipulator joints is controlled by a weight matrix, which is constant and related to each link’s energy consumption. To dynamically adjust the weight matrix according to kinematic constraints and acquire better energy efficiency, an adaptive WNI (AWNI) algorithm is further proposed. In AWNI, the weight matrix is adjusted in real-time during the planning process, with considerations of the kinematic constraints and the energy consumption of the manipulator. The switch of the links between the working state and the non-working state is made through the weight matrix to achieve flexible control of the manipulator motion. Two evaluation functions are established to validate the effectiveness of AWNI in energy saving and motion stability control. Taking a 6 degrees of freedom (DOF) manipulator as an example, simulation experiments on trajectory planning are carried out and the results show the effectiveness of the proposed AWNI algorithm.

Accurate and robust 6-DOF (6D) pose estimation from a single RGB-D image is an essential task of intelligent manufacturing, such as robot assembly and digital twin. However, incomplete and noisy 3D data acquired from depth sensors make the task challenging, especially for various industrial parts without sufficient textures, where the occlusion further exacerbates the problem. To tackle this issue, this paper proposes a geometry-enhanced network with incomplete shape recovery (GER-Net) to estimate the 6D pose of industrial parts. Firstly, an incomplete 3D shape recovery (ISR) module with a learnable Shape Protection (SP) Layer is introduced to recover the complete 3D geometry shapes of raw point clouds obtained from depth measurements. Subsequently, the multi-modal features extracted from raw RGB-D data are enhanced with the geometry information from the recovered point cloud via multi-scale concatenation and recurrent forward fusion in the point cloud space. In this way, the enhanced RGB-D representations contribute to the regression of accurate 6D pose. Experiments on two popular benchmark datasets (LineMOD and Occlusion-LineMOD) show that the proposed approach achieves state-of-the-art performance. Furthermore, a real-world low-texture industrial part dataset ITM3D-Parts is presented to fully validate the effectiveness of our method, where it also achieves the best performance with remarkable accuracy and robustness.

Accurate 3D object segmentation in point clouds is a basis for industrial robot applications, such as robot manipulation and digital twin, which require an understanding of the 3D environment. However, the unstructured and disordered nature of point clouds makes it challenging, especially for the incomplete 3D data under a single view in the real-world scenario. To this end, this paper proposes a novel 3D object segmentation framework (3DT-Seg) based on Cross-Window Point Transformer (CP-Former). CP-Former captures the long-range dependencies between local windows and latent semantic boundaries to enhance the point-wise features extracted from irregular point clouds via a bidirectional cross-attention mechanism. In addition, a contrastive learning loss and an adaptive dual aggregation strategy are introduced on semantic transition regions during the semantic supervising and instance clustering process, respectively. In this way, the latent boundary information is further utilized to improve the overall segmentation performance. Experiments on the popular benchmark (SI3DS) dataset show the state-of-the-art performance of the proposed approach in terms of semantic and instance segmentation. Furthermore, a real-world point cloud dataset (IP-Cloud) for the robotic grasping task is presented to fully validate the effectiveness of our method in practice, where it also achieves remarkable performance.