Indoor autonomous navigation refers to the perception and exploration abilities of mobile agents in unknown indoor environments with the help of various sensors. It is the basic and one of the most important functions of mobile agents. In spite of the high performance of the single-sensor navigation method, multi-sensor fusion methods still potentially improve the perception and navigation abilities of mobile agents. This work summarizes the multi-sensor fusion methods for mobile agents’ navigation by: (1) analyzing and comparing the advantages and disadvantages of a single sensor in the task of navigation; (2) introducing the mainstream technologies of multi-sensor fusion methods, including various combinations of sensors and several widely recognized multi-modal sensor datasets. Finally, we discuss the possible technique trends of multi-sensor fusion methods, especially its technique challenges in practical navigation environments.

In recent years, increasing image data comes from various sensors, and object detection plays a vital role in image understanding. For object detection in complex scenes, more detailed information in the image should be obtained to improve the accuracy of detection task. In this paper, we propose an object detection algorithm by jointing semantic segmentation (SSOD) for images. First, we construct a feature extraction network that integrates the hourglass structure network with the attention mechanism layer to extract and fuse multi-scale features to generate high-level features with rich semantic information. Second, the semantic segmentation task is used as an auxiliary task to allow the algorithm to perform multi-task learning. Finally, multi-scale features are used to predict the location and category of the object. The experimental results show that our algorithm substantially enhances object detection performance and consistently outperforms other three comparison algorithms, and the detection speed can reach real-time, which can be used for real-time detection.

Nowadays, robots have been widely used in various manufacturing assemblies. However, in the mobile phone flexible printed circuit (FPC) assembly task, robots are still far from trustworthy because the connector and receiver of FPC are very small in size, and detection for them is influenced by the partial occlusion of the robot end manipulator in operation. This makes it difficult to measure the spatial relationship between the connector and receiver in robot picking and buckling. To this end, this work proposes a Digit Twin (DT)-driven measurement method for partial occlusion FPC assembly. The contributions include: (1) we introduce a homography-based strategy to calibrate the FPC placement panel and its virtual placement panel in digit twin. It makes the partial occlusion views in physical space visible in digital twin within tolerable errors. This helps to improve the measurement by eliminating the observation occlusion which originally exits in physical space; (2) we propose a DT-driven reward function measured by the distance and rotation difference between the FPC connector hold by the manipulator and its ideal position in the digital twin. This makes the deep architecture can learn the actions guided by the measurements of the FPC positions in physical space and digital twin simultaneously. The experiments are carried out on a practical assembly platform for the mobile phone Redmi Note 11. With the number of failed trails limited, the proposed network obviously outperforms the pure vision-based measurement and the network without DT support. The proposed method provides a primitive but novel measurement strategy for mobile phone FPC assembly. It can be applied to the real mobile phone assembly lines to reduce the labor burden in the near future.

The traditional Deep Deterministic Policy Gradient (DDPG) algorithm frequently exhibits a notable reduction in success rate when transferred to new environments after being trained in complex simulation settings. To address these issues, this paper adopts a Multi-Environment (Multi-Env) parallel training approach and integrates Multi-Head Attention (MHA) and Prioritized Experience Replay (PER) into the DDPG framework, optimizing the reward function to form the MAP-DDPG algorithm. This approach enhances the algorithm’s generalization capability and execution efficiency. Through comparative training and testing of the DDPG and MAP-DDPG algorithms in both simulation and real-world environments, the experimental results demonstrate that MAP-DDPG significantly improves generalization and execution efficiency over the DDPG algorithm. Specifically, in simulation environment tests, the MAP-DDPG algorithm achieved an average 30% increase in success rate and reduced the average time to reach the target point by 23.7 s compared to the DDPG algorithm. These results indicate that the MAP-DDPG algorithm significantly enhances path planning generalization and execution efficiency, providing a more effective solution for path planning in complex environments.

Real-time 3D reconstruction is one of the current popular research directions of computer vision, and it has become the core technology in the fields of virtual reality, industrialized automatic systems, and mobile robot path planning. Currently, there are three main problems in the real-time 3D reconstruction field. Firstly, it is expensive. It requires more varied sensors, so it is less convenient. Secondly, the reconstruction speed is slow, and the 3D model cannot be established accurately in real time. Thirdly, the reconstruction error is large, which cannot meet the requirements of scenes with accuracy. For this reason, we propose a real-time 3D reconstruction method based on monocular vision in this paper. Firstly, a single RGB-D camera is used to collect visual information in real time, and the YOLACT++ network is used to identify and segment the visual information to extract part of the important visual information. Secondly, we combine the three stages of depth recovery, depth optimization, and deep fusion to propose a three-dimensional position estimation method based on deep learning for joint coding of visual information. It can reduce the depth error caused by the depth measurement process, and the accurate 3D point values of the segmented image can be obtained directly. Finally, we propose a method based on the limited outlier adjustment of the cluster center distance to optimize the three-dimensional point values obtained above. It improves the real-time reconstruction accuracy and obtains the three-dimensional model of the object in real time. Experimental results show that this method only needs a single RGB-D camera, which is not only low cost and convenient to use, but also significantly improves the speed and accuracy of 3D reconstruction.

In the industrial assembly of flexible printed circuits (FPCs), the FPC connectors and receivers are required to connect successfully with the least possible buckling times. However, precise and autonomous positioning of the FPC connectors is still a great challenge, as they are very small and visually blocked during assembly. This article proposes a strategy for the industrial assembly of FPC by: 1) unlike the traditional vision based methods, the task of FPC position identification in this work is considered as a haptic signals alignment problem. We mount a haptic sensor at the top end of the robotic arm manipulator and record various mismatched haptic signals around the ideal FPC connector position; 2) converting both the correct and incorrect haptic signals into images; and 3) we introduce a novel deep Siamese-network (DSN) structure to align the incorrect haptic images to the ground truth. The experiments are carried out on a practical FPC assembly platform for the Redmi Note 11 mobile phone. It is found that the haptic images combined with DSN can significantly improve the FPC location accuracy. When the buckle times are limited to 10, 5, and 1, the proposed method outperforms the state-of-the-art methods in a realistic FPC assembly platform.