The 3D semantic map plays an increasingly important role in a wide variety of applications, especially for many kinds of task-driven robots. In this paper, we present a semantic mapping methodology for 3D semantic map obtaining from RGB-D scans. In contrast to existing methods that use 3D annotated information as supervisory, we focus on accurate 2D frame labeling and combine labels in 3D space using semantic fusion mechanism. For scene parsing, a two-stream network with a novel discriminatory mask loss is proposed to explore sufficient extraction and fusion of RGB and depth information achieving steadily semantic segmentation. The discriminatory mask guides the cross-entropy loss function and interprets the influence of different pixels on back-propagation, which reduces the harmful effects of the depth noise or the fallible annotation at the edges of objects. After the correspondences between frames are provided, these semantic frames are fused in unified 3D coordinates using the novel label-oriented voxelgrid filter. It can ensure the intra-frame spatial continuity and the inter-frame spatiotemporal consistency through introducing the label-oriented statistical principle into labeled point clouds. In order to avoid the unfavorable interference between uncorrelated frames, we further propose an adaptive grouping algorithm by applying the view frustum filter to group frames with sufficient overlap as a segment. To this end, we demonstrate the effectiveness of the proposed method on the 2D/3D semantic label benchmark of ScanNetv2 and Cityscapes datasets.

We propose a deep-learning-based approach to producing computer-generated holograms (CGHs) of real-world scenes. We design an end-to-end convolutional neural network (the Stereo-to-Hologram Network, SHNet) framework that takes a stereo image pair as input and efficiently synthesizes a monochromatic 3D complex hologram as output. The network is able to rapidly and straightforwardly calculate CGHs from the directly recorded images of real-world scenes, eliminating the need for time-consuming intermediate depth recovery and diffraction-based computations. We demonstrate the 3D reconstructions with clear depth cues obtained from the SHNet-based CGHs by both numerical simulations and optical holographic virtual reality display experiments.

Visual-inertial SLAM systems achieve highly accurate estimation of camera motion and 3D representation of the environment. Most of the existing methods rely on points by feature matching or direct image alignment using photo-consistency constraints. The performance of these methods usually decreases when facing low textured environments. In addition, lines are also very common in man-made environments and provide geometrical structure information of the environment. In this paper, we increase the robustness of visual-inertial SLAM system to handle these situations by using both points and lines. Our method, implemented based on ORB-SLAM2, makes the combination of points, lines and IMU measurements in an effective way by selecting keyframes very carefully and handling the outlier lines efficiently. The cost function of bundle adjustment is formed by point, line reprojection errors and IMU residual errors. We derive the Jacobian matrices of line reprojection errors with respect to the 3D endpoints of line segments and camera motion. Loop closure detection is decided by both point and line features using the bag-of-words approach. Our method is evaluated on the public EuRoc dataset, and compared with the state-of-the-art visual-inertial fusion methods. Experimental results show that our method achieves the highest accuracy on most of testing sequences, especially in some challengeable situations such as low textured and illumination changing environments.

Visual Simultaneously Localization and Mapping (SLAM) systems obtain accurate ego-motion of cameras and 3D representation of the environment. Although the performance of SLAM systems is very impressive in many common scenes, tracking failure is still a very challenging issue, which always exist in low-textured environment and rapid camera motion situations. In this paper, we propose a real-time stereo visual SLAM system using bionic eyes. The new type sensor named bionic eyes are basically the stereo cameras and perform all the movements of human eyes. Our system makes a good combination of sensors and algorithms. In order to find more feature points, we proposed an autonomous searching strategy of the bionic eyes, which is mainly inspired by the peripheral and central visual of human eyes. After that the camera motion is calculated using a robust motion estimation method. Our system is tested in indoor low-textured and large-scale outdoor environment. Experimental results show that compared to fixed stereo cameras, our bionic eyes SLAM system gains more robustness and avoid tracking failure problem when facing the low-textured environment.

Perspective-n-Point (PnP) problem aims to estimate pose from known 3D map points and their projections. Efficient PnP (EPnP), one of the classical PnP solvers, represents camera pose with control points, which are easier to estimate utilizing the least square (LS) formulation. However, the geometry refinement procedure performed by most EPnP-based methods is separated from the solution of LS formulation, which creates difficulty in balancing between minimizing the loss function and preserving the essential geometry properties of control points. To handle the problem, we propose a novel method to integrate geometry constraints into control points formulation and reformulate the LS to quadratic constraints quadratic programming (QCQP). We deduce an innovative analytical solution, dubbed ACEPnP, to the constrained EPnP problem, which is faster than the customarily applied numerical methods. An uncertainty-aware least square registration procedure is designed in ACEPnP to compute camera pose from control points. Our method is plug-and-play and can be embedded in various variants of EPnP. Experiments in synthetic and real data show that our methods outperform other state-of-the-arts. Compared to the best PnP solver, EPnPU*, our approaches reduce rotation errors by 12.2% and translation errors by 34.8% with similar time consumption on the KITTI odometry dataset.