As one of the typical application-oriented solutions to robot autonomous navigation, visual simultaneous localization and mapping is essentially restricted to simplex environmental understanding based on geometric features of images. By contrast, the semantic simultaneous localization and mapping that is characterized by high-level environmental perception has apparently opened the door to apply image semantics to efficiently estimate poses, detect loop closures, build 3D maps, and so on. This article presents a detailed review of recent advances in semantic simultaneous localization and mapping, which mainly covers the treatments in terms of perception, robustness, and accuracy. Specifically, the concept of “semantic extractor” and the framework of “modern visual simultaneous localization and mapping” are initially presented. As the challenges associated with perception, robustness, and accuracy are being stated, we further discuss some open problems from a macroscopic view and attempt to find answers. We argue that multiscaled map representation, object simultaneous localization and mapping system, and deep neural network-based simultaneous localization and mapping pipeline design could be effective solutions to image semantics-fused visual simultaneous localization and mapping.

For robots in unknown environments, the visual-inertial simultaneous localization and mapping (SLAM) incorporates the multisensor data and enables a globally consistent trajectory tracking and mapping. Given the scene structures in the SLAM problem, the high-quality lines can serve as primitives and provide additional constraints for camera pose estimation. This study is devoted to a pipeline design of dynamically fused point-line visual-inertial navigation system (VINS). The design builds on the heritage of VINS-Mono but employs an ensemble pose estimation strategy to integrate point and line features. As far as we know, it is the first optimization-based monocular VINS method that gives an insight into the quantitative relationship between point and line feature numbers. Additionally, the measured pixels for line feature-length determination and two hidden parameters in line segment detector (LSD) extractor are investigated, guaranteeing the line feature extraction optimization in both quality and efficiency. The benchmark dataset and real-world tests are conducted. In terms of challenging scene tests in the EuRoc dataset, results are compared against ROVIO, VINS-Mono, and PL-VINS frameworks, revealing our design is substantially more accurate and robust than others with both locally and globally consistent pose estimates. The engineered algorithm in the camera “Mynteye” further enables the Bulldog-CX Robot platform verification. The results concerning the drifts, time taken for line feature extraction, and real-world trajectory consistency show our design outperforms the point-based VINS. The strong adaptability of our dynamically fused point-line VINS to the broad range of indoor and outdoor scenes is simultaneously verified.

Audiovisual speech recognition is a favorable solution to multimodality human–computer interaction. For a long time, it has been very difficult to develop machines capable of generating or understanding even fragments of natural languages; the fused sight, smelling, touching, and so on provide machines with possible mediums to perceive and understand. This article presents a detailed review of recent advances in audiovisual speech recognition area. After explicitly representing audiovisual speech recognition development phase divided by timeline, we focus on typical audiovisual speech database descriptions in terms of single view and multi-view, since the public databases for general purpose should be the first concern for audiovisual speech recognition tasks. For the following challenges that are inseparably related to the feature extraction and dynamic audiovisual fusion, the principal usefulness of deep learning-based tools, such as deep fully convolutional neural network, bidirectional long short-term memory network, 3D convolutional neural network, and so on, lies in the fact that they are relatively simple solutions of such problems. As the principle analyses and comparisons related to computational load, accuracy, and applicability of well-developed audiovisual speech recognition frameworks have been conducted, we further illuminate our insights into the future audiovisual speech recognition architecture design. We argue that end-to-end audiovisual speech recognition model and deep learning-based feature extractors will guide multimodality human–computer interaction directly to a solution.

Polarized skylight is as fundamental a constituent of passive navigation as the geomagnetic field. With regard to its applicability to outdoor robot localization, a polarized light-aided visual-inertial navigation system (VINS) modelization dedicated to globally optimized pose estimation and heading correction is constructed. The combined system follows typical visual simultaneous localization and mapping (SLAM) frameworks, and we propose a methodology to fuse global heading measurements with visual and inertial information in a graph optimization-based estimator. With ideas of‘adding new attributes of graph vertices and creating heading error-encoded constraint edges’, the heading, as the absolute orientation reference, is estimated by the Berry polarization model and continuously updated in a graph structure. The formulized graph optimization process for multi-sensor fusion is simultaneously provided. In terms of campus road experiments on the Bulldog-CX robot platform, the results are compared against purely stereo camera-dependent and VINS Fusion frameworks, revealing that our design is substantially more accurate than others with both locally and globally consistent position and attitude estimates. As a passive and tightly coupled navigation mode, the polarized light-aided VINS can therefore be considered as a tool candidate for a class of visual SLAM-based multi-sensor fusion.

An adaptive Genetic Algorithm (GA) is proposed, which focuses on the automatic adjustments of crossover probability and mutation probability with the changeable environmental parameters. The improved algorithm can overcome some disadvantages of traditional GA, such as, early falling into local optimum, lower convergence speed and large calculation etc. In sequence, the complementary characteristic between crossover probability and mutation probability is obtained through carrying out the numerical simulation. The results demonstrate that, compared with the traditional GA, the adaptive one leads to better performance in path curves and fitness, when 30 generations operations is implemented. This solution mentioned above, is proved to a better choice for practical application in path planning for mobile robots.

The development and maturation of simultaneous localization and mapping (SLAM) in robotics opens the door to the application of a visual inertial odometry (VIO) to the robot navigation system. For a patrol robot with no available Global Positioning System (GPS) support, the embedded VIO components, which are generally composed of an Inertial Measurement Unit (IMU) and a camera, fuse the inertial recursion with SLAM calculation tasks, and enable the robot to estimate its location within a map. The highlights of the optimized VIO design lie in the simplified VIO initialization strategy as well as the fused point and line feature-matching based method for efficient pose estimates in the front-end. With a tightly-coupled VIO anatomy, the system state is explicitly expressed in a vector and further estimated by the state estimator. The consequent problems associated with the data association, state optimization, sliding window and timestamp alignment in the back-end are discussed in detail. The dataset tests and real substation scene tests are conducted, and the experimental results indicate that the proposed VIO can realize the accurate pose estimation with a favorable initializing efficiency and eminent map representations as expected in concerned environments. The proposed VIO design can therefore be recognized as a preferred tool reference for a class of visual and inertial SLAM application domains preceded by no external location reference support hypothesis.