The 3D semantic map plays an increasingly important role in a wide variety of applications, especially for many kinds of task-driven robots. In this paper, we present a semantic mapping methodology for 3D semantic map obtaining from RGB-D scans. In contrast to existing methods that use 3D annotated information as supervisory, we focus on accurate 2D frame labeling and combine labels in 3D space using semantic fusion mechanism. For scene parsing, a two-stream network with a novel discriminatory mask loss is proposed to explore sufficient extraction and fusion of RGB and depth information achieving steadily semantic segmentation. The discriminatory mask guides the cross-entropy loss function and interprets the influence of different pixels on back-propagation, which reduces the harmful effects of the depth noise or the fallible annotation at the edges of objects. After the correspondences between frames are provided, these semantic frames are fused in unified 3D coordinates using the novel label-oriented voxelgrid filter. It can ensure the intra-frame spatial continuity and the inter-frame spatiotemporal consistency through introducing the label-oriented statistical principle into labeled point clouds. In order to avoid the unfavorable interference between uncorrelated frames, we further propose an adaptive grouping algorithm by applying the view frustum filter to group frames with sufficient overlap as a segment. To this end, we demonstrate the effectiveness of the proposed method on the 2D/3D semantic label benchmark of ScanNetv2 and Cityscapes datasets.

We present a simple yet highly efficient method to eliminate spurious trails of dynamic objects for 3D point cloud map updating. Firstly, we extract the view overlaps based on view frustum filter. Then, we obtain spurious trails via bidirectional searching of view overlaps using a KD tree. Finally, in terms of the situation where moving objects occlude part of background due to the limits of the RGB-D camera, we design a ray tracing principle based filter to supplement the missing background in the whole point cloud map. Our method can be integrated into any SLAM or 3D reconstruction systems with RGB-D data input and it is suitable for both static and dynamic environments. We validate our approach in realworld scenes of our laboratory office using a Kinect system for robot obstacles avoidance and navigation. Moreover, experiments on the KITTI odometry benchmark illustrate that the proposed approach is highly efficient for dynamic spurious trails rejection and 3D map updating.

Very deep convolutional neural networks (CNNs) have recently achieved great success in stereo matching. It is still highly desirable to learn a robust feature map to improve ill-posed regions, such as weakly textured regions, reflective surfaces, and repetitive patterns. Therefore, we propose an end-to-end multi-dimensional residual dense attention network (MRDA-Net) in this paper, focusing on more comprehensive pixel-wise feature extraction. Our proposed network consists of two parts: the 2D residual dense attention net for feature extraction and the 3D convolutional attention net for matching. Our designed 2D residual dense attention net uses a dense network structure to fully exploit the hierarchical features from preceding convolutional layers and uses residual network structure to fuse low-level structure information and high-level semantic information. The 2D attention module of the net aims to adaptively recalibrate channel-wise features to be more concerned about informative features. Our proposed 3D convolutional attention net further expands attention mechanism for matching. The stacked hourglass module of the net is exploited to extract multi-scale context information as well as geometry information. The novel 3D attention module of the net aggregates hierarchical sub-cost volumes adaptively instead of manually and then achieves a comprehensive recalibrated cost volume for more correct disparity computation. The experiments demonstrate that our approach achieves the state-of-the-art accuracy on Scene Flow dataset and KITTI 2012 and KITTI 2015 Stereo datasets.

Scene understanding is one of the foundations for robots to achieve true artificial intelligence. Semantic segmentation that imitates the mechanism of human visual system can effectively promote the correctness of scene understanding. It conforms to the basic principle of human environment perception and enables robots to better serve human society. In this paper, we propose a multilevel cross-aware network (MCA-Net) for RGBD semantic segmentation. It utilizes basic residual structure to encode texture information and depth geometric information respectively. In view of the flow pattern of different visual features in the visual pathway of human brain, our MCA-Net attempts to jointly reason about 2D appearance and depth geometric information. Therefore, multilevel cross-aware fusion modules are designed to fuse multi-scale complementary features extracted from RGB and depth images. As the depth and color information are independent, the reasonable combination of depth and RGB images can improve the quality of semantic labeling. Experiments conducted on ScanNetv2 dataset show that the proposed network produces high quality segmentation results of RGBD images and outperforms state-of-the-art methods. Furthermore, the semantic labeling results on the bionic binocular robot in real-office scenes further demonstrate the effectiveness of the proposed MCA-Net.

Microscopic profilometry based on fringe projection is notable for its noncontact and robust nature and is widely used in industrial electronics. However, its ability to observe electronic packages is limited by the depth of field (DOF) of the lens. This article proposes a novel reconstruction scheme based on telecentric 3-D profilometry to overcome this limitation. The advantage of profilometry based on phase-shifting fringe projection is utilized through defocus detection to achieve high-precision contour measurement with a single DOF. After completing such local reconstruction, a complete 3-D model is registered from partial fragments in different DOFs, making it simple for the microscope system to measure the geometric structure of objects whose thickness exceeds three times the DOF. To accommodate occlusion in local reconstruction, a specific detection algorithm is designed based on intensity variance. Beyond that, we present a novel projection system calibration scheme to improve the calibration robustness and accuracy. Instead of directly using the detected phase to calculate the coordinates of the corners, we first model the relationship between the phase and pixel coordinates and then leverage the estimated parameters to recover the phase. Our proposed method reduces the influence of calibration board spraying errors on the calibration results. Experiments show the effectiveness of this new approach.

Disparity calculation is crucial for binocular sensor ranging. The disparity estimation based on edges is an important branch in the research of sparse stereo matching and plays an important role in visual navigation. In this paper, we propose a robust sparse stereo matching method based on the semantic edges. Some simple matching costs are used first, and then a novel adaptive dynamic programming algorithm is proposed to obtain optimal solutions. This algorithm makes use of the disparity or semantic consistency constraint between the stereo images to adaptively search parameters, which can improve the robustness of our method. The proposed method is compared quantitatively and qualitatively with the traditional dynamic programming method, some dense stereo matching methods, and the advanced edge-based method respectively. Experiments show that our method can provide superior performance on the above comparison.