In this paper, a simultaneous localization and mapping (SLAM) method that eliminates the influence of moving objects in dynamic environments is proposed. This method utilizes the correlation between map points to separate points that are part of the static scene and points that are part of different moving objects into different groups. A sparse graph is first created from all map points. In this graph, the vertices represent map points, and each edge represents the correlation between adjacent points. If the relative position between two points remains consistent over time, there is correlation between them, and they are considered to be moving together rigidly. If not, they are considered to have no correlation and to be in separate groups. After the edges between the uncorrelated points are removed, the remaining graph separates the map points of moving objects from the static scene. The largest group is assumed to be the group of reliable static map points. Experimental results demonstrate that robust and accurate performance can be achieved by the proposed SLAM method in both slightly and highly dynamic environments. Compared to other state-of-the-art methods, the proposed method can provide competitive accuracy with good real-time performance.

Neural network models of machine learning have shown promising prospects for visual tasks, such as facial emotion recognition (FER). However, the generalization of the model trained from a dataset with a few samples is limited. Unlike the machine, the human brain can effectively realize the required information from a few samples to complete the visual tasks. To learn the generalization ability of the brain, in this article, we propose a novel brain-machine coupled learning method for facial emotion recognition to let the neural network learn the visual knowledge of the machine and cognitive knowledge of the brain simultaneously. The proposed method utilizes visual images and electroencephalogram (EEG) signals to couple training the models in the visual and cognitive domains. Each domain model consists of two types of interactive channels, common and private. Since the EEG signals can reflect brain activity, the cognitive process of the brain is decoded by a model following reverse engineering. Decoding the EEG signals induced by the facial emotion images, the common channel in the visual domain can approach the cognitive process in the cognitive domain. Moreover, the knowledge specific to each domain is found in each private channel using an adversarial strategy. After learning, without the participation of the EEG signals, only the concatenation of both channels in the visual domain is used to classify facial emotion images based on the visual knowledge of the machine and the cognitive knowledge learned from the brain. Experiments demonstrate that the proposed method can produce excellent performance on several public datasets. Further experiments show that the proposed method trained from the EEG signals has good generalization ability on new datasets and can be applied to other network models, illustrating the potential for practical applications.

Multimodal Sentiment Analysis is a promising research area for modeling multiple heterogeneous modalities. Two major challenges that exist in this area are a) multimodal data is unaligned in nature due to the different sampling rates of each modality, and b) long-range dependencies between elements across modalities. These challenges increase the difficulty of conducting efficient multimodal fusion. In this work, we propose a novel end-to-end network named Cross Hyper-modality Fusion Network (CHFN). The CHFN is an interpretable Transformer-based neural model that provides an efficient framework for fusing unaligned multimodal sequences. The heart of our model is to dynamically adjust word representations in different non-verbal contexts using unaligned multimodal sequences. It is concerned with the influence of non-verbal behavioral information at the scale of the entire utterances and then integrates this influence into verbal expression. We conducted experiments on both publicly available multimodal sentiment analysis datasets CMU-MOSI and CMU-MOSEI. The experiment results demonstrate that our model surpasses state-of-the-art models. In addition, we visualize the learned interactions between language modality and non-verbal behavior information and explore the underlying dynamics of multimodal language data.

In recent years, longwave infrared (LWIR) cameras have become potential in visual simultaneous localization and mapping (SLAM) research since the delivered thermal images can provide information beyond the visible spectrum and are robust to environment illumination. However, due to modality differences, SLAM methods designed for visible cameras cannot be directly applied to thermal data. In this paper, we propose a thermal-inertial SLAM method for all-day autonomous systems. To overcome the challenge of the thermal data association, the proposed method represents several improvements, including singular-value-decomposition-based (SVD-based) image processing and ThermalRAFT tracking methods. Based on the characteristics of the thermal images, the SVD-based image processing method can exploit the fixed noise pattern of thermal images and enhance the image quality to improve the performance of subsequent steps, including thermal feature extraction and loop detection. To achieve real-time and robust feature tracking, we develop ThermalRAFT, an efficient optical flow network with iterative optimization. Moreover, the system introduces a bag-of-words-based loop detection method to maintain global consistency in long-term operation. The experimental results demonstrate that the proposed method can provide competitive performance in indoor and outdoor environments and is robust under challenging illumination conditions.

Since visible cameras rely on ideal illumination to provide adequate environment information, visual simultaneous localization and mapping (SLAM) under extreme illumination remains a challenge. Therefore, we propose a direct near-infrared-depth visual SLAM method with an active near-infrared (NIR) light source. The NIR light source provides necessary lighting conditions without interference to human activities. The proposed method utilizes a direct bundle adjustment method jointly fusing near-infrared and depth images to optimize camera motion. In the front-end tracking, the albedo transformed images and depth maps are used in a direct image alignment. During the back-end optimization, the direct bundle adjustment method exploits the albedo-consistency clues to minimize the geometry and albedo residuals. The indoor experiments show that the proposed method can provide robust performances without being affected by visible illumination. Compared with leading SLAM methods using active light source, the proposed method still offers competitive and robust results. The results also show that NIR-based direct SLAM outperforms the same method based on visible light, indicating that the NIR-based implementation is a potential solution for robust pose estimation indoors.

Light Detection and Ranging (LiDAR) systems are novel sensors that provide robust distance and reflection strength by active pulsed laser beams. They have significant advantages over visual cameras by providing active depth and intensity measurements that are robust to ambient illumination. However, the systemsstill pay limited attention to intensity measurements since the output intensity maps of LiDAR sensors are different from conventional cameras and are too sparse. In this work, we propose exploiting the information from both intensity and depth measurements simultaneously to complete the LiDAR intensity maps. With the completed intensity maps, mature computer vision techniques can work well on the LiDAR data without any specific adjustment. We propose an end-to-end convolutional neural network named LiDAR-Net to jointly complete the sparse intensity and depth measurements by exploiting their correlations. For network training, an intensity fusion method is proposed to generate the ground truth. Experiment results indicate that intensity–depth fusion can benefit the task and improve performance. We further apply an off-the-shelf object (lane) segmentation algorithm to the completed intensity maps, which delivers consistent robust to ambient illumination performance. We believe that the intensity completion method allows LiDAR sensors to cope with a broader range of practice applications.