Many scholars have proposed different single-robot coverage path planning (SCPP) and multi-robot coverage path planning (MCPP) algorithms to solve the coverage path planning (CPP) problem of robots in specific areas. However, in outdoor environments, especially in emergency search and rescue tasks, complex geographic environments reduce the task execution efficiency of robots. Existing CPP algorithms have hardly considered environmental complexity. This paper proposed an MCPP algorithm considering the complex land cover types in outdoor environments to solve the related problems. The algorithm first describes the visual fields of the robots in different land cover types by constructing a hierarchical quadtree and builds the adjacent topological relations among the cells in the same and different layers in the hierarchical quadtree by defining shared neighbor direction based on Binary System. The algorithm then performs an approximately balanced task assignment to the robots considering the moving speeds in different land cover types using the azimuth trend method we proposed to ensure the convergence of the task assignment process. Finally, the algorithm improves Spanning Tree Covering (STC) algorithm to complete the CPP in the area where each robot belongs. This study used a classification image of the real outdoor environment to the verification of the algorithm. Results show that the coverage paths planned by the algorithm are reasonable and efficient and its performance has obvious advantages compare with the current mainstream MCPP algorithm.

Object detection plays an important role in autonomous driving, disaster rescue, robot navigation, intelligent video surveillance, and many other fields. Nonetheless, visible images are poor under weak illumination conditions, and thermal infrared images are noisy and have low resolution. Consequently, neither of these two data sources yields satisfactory results when used alone. While some scholars have combined visible and thermal images for object detection, most did not consider the illumination conditions and the different contributions of diverse data sources to the results. In addition, few studies have made use of the temperature characteristics of thermal images. Therefore, in the present study, visible and thermal images are utilized as the dataset, and RetinaNet is used as the baseline to fuse features from different data sources for object detection. Moreover, a dynamic weight fusion method, which is based on channel attention according to different illumination conditions, is used in the fusion component, and the channel attention and a priori temperature mask (CAPTM) module is proposed; the CAPTM can be applied to a deep learning network as a priori knowledge and maximizes the advantage of temperature information from thermal images. The main innovations of the present research include the following: (1) the consideration of different illumination conditions and the use of different fusion parameters for different conditions in the feature fusion of visible and thermal images; (2) the dynamic fusion of different data sources in the feature fusion of visible and thermal images; (3) the use of temperature information as a priori knowledge (CAPTM) in feature extraction. To a certain extent, the proposed methods improve the accuracy of object detection at night or under other weak illumination conditions and with a single data source. Compared with the state-of-the-art (SOTA) method, the proposed method is found to achieve superior detection accuracy with an overall mean average precision (mAP) improvement of 0.69%, including an AP improvement of 2.55% for the detection of the Person category. The results demonstrate the effectiveness of the research methods for object detection, especially temperature information-rich object detection.

Automatically translating chromaticity-free thermal infrared (TIR) images into realistic color visible (CV) images is of great significance for autonomous vehicles, emergency rescue, robot navigation, nighttime video surveillance, and many other fields. Most recent designs use end-to-end neural networks to translate TIR directly to CV; however, compared to these networks, TIR has low contrast and an unclear texture for CV translation. Thus, directly translating the TIR temperature value of only one channel to the RGB color value of three channels without adding additional constraints or semantic information does not handle the one-to-three mapping problem between different domains in a good way, causing the translated CV images not only to have blurred edges but also color confusion. As for the methodology of the work, considering that in the translation from TIR to CV the most important process is to map information from the temperature domain into the color domain, an improved CycleGAN (GMA-CycleGAN) is proposed in this work in order to translate TIR images to grayscale visible (GV) images. Although the two domains have different properties, the numerical mapping is one-to-one, which reduces the color confusion caused by one-to-three mapping when translating TIR to CV. Then, a GV-CV translation network is applied to obtain CV images. Since the process of decomposing GV images into CV images is carried out in the same domain, edge blurring can be avoided. To enhance the boundary gradient between the object (pedestrian and vehicle) and the background, a mask attention module based on the TIR temperature mask and the CV semantic mask is designed without increasing the network parameters, and it is added to the feature encoding and decoding convolution layers of the CycleGAN generator. Moreover, a perceptual loss term is applied to the original CycleGAN loss function to bring the translated images closer to the real images regarding the space feature. In order to verify the effectiveness of the proposed method, the FLIR dataset is used for experiments, and the obtained results show that, compared to the state-of-the-art model, the subjective quality of the translated CV images obtained by the proposed method is better, as the objective evaluation metric FID (Fréchet inception distance) is reduced by 2.42 and the PSNR (peak signal-to-noise ratio) is improved by 1.43.

This paper presents a method for mosaicking unmanned aerial vehicle (UAV) multispectral images. The main purpose of the proposed method is to reduce spatial distortion in the mosaicking process and increase robustness and the speed of the operation. Most UAV multispectral images have multiple bands, and in every band, ground targets have a variety of reflection characteristics that will result in diverse feature quality for feature matching. In this research, an information entropy-based evaluation method is used to select the optimal band for feature matching among the UAV images. To produce more robust matching results for the following alignment step, the evaluation method takes the contrast and spatial distribution of the feature points into consideration at the same time. In most common image mosaicking processes, the digital orthophoto map (DOM) is generated to achieve maximum spatial accuracy. During this process, the original image data will experience considerable irregular resampling, and the process is also unstable in some circumstances. The alignment step uses a simplified projection model that treats the ground as planar is provided, by which the alignment parameters are applied directly to the images instead of generating 3D points, to avoid irregular resampling and unstable 3D reconstruction. The proposed method is proved to be more efficient and accurate and has lower spectral distortion than state-of-the-art mosaicking software.

Nighttime Thermal InfraRed (NTIR) image colorization, also known as the translation of NTIR images into Daytime Color Visible (DCV) images, can facilitate human and intelligent system perception of nighttime scenes under weak lighting conditions. End-to-end neural networks have been used to learn the mapping relationship between temperature and color domains, and translate NTIR images with one channel into DCV images with three channels. However, this mapping relationship is an ill-posed problem with multiple solutions without constraints, resulting in blurred edges, color disorder, and semantic errors. To solve this problem, an NTIR2DCV method that includes two steps is proposed: firstly, fuse Nighttime Color Visible (NCV) images with NTIR images based on an Illumination-Aware, Multilevel Decomposition Latent Low-Rank Representation (IA-MDLatLRR) method, which considers the differences in illumination conditions during image fusion and adjusts the fusion strategy of MDLatLRR accordingly to suppress the adverse effects of nighttime lights; secondly, translate the Nighttime Fused (NF) image to DCV image based on HyperDimensional Computing Generative Adversarial Network (HDC-GAN), which ensures feature-level semantic consistency between the source image (NF image) and the translated image (DCV image) without creating semantic label maps. Extensive comparative experiments and the evaluation metrics values show that the proposed algorithms perform better than other State-Of-The-Art (SOTA) image fusion and translation methods, such as FID and KID, which decreased by 14.1 and 18.9, respectively.

This paper presents an augmented reality-based method for geo-registering videos from low-cost multi-rotor Unmanned Aerial Vehicles (UAVs). The goal of the proposed method is to conduct an accurate geo-registration and target localization on a UAV video stream. The geo-registration of video stream requires accurate attitude data. However, the Inertial Measurement Unit (IMU) sensors on most low-cost UAVs are not capable of being directly used for geo-registering the video. The magnetic compasses on UAVs are more vulnerable to the interferences in the working environment than the accelerometers. Thus the camera yaw error is the main sources of the registration error. In this research, to enhance the low accuracy attitude data from the onboard IMU, an extended Kalman Filter (EKF) model is used to merge Real Time Kinematic Global Positioning System (RTK GPS) data with the IMU data. In the merge process, the high accuracy RTK GPS data can be used to promote the accuracy and stability of the 3-axis body attitude data. A method of target localization based on the geo-registration model is proposed to determine the coordinates of the ground targets in the video. The proposed method uses a modified extended Kalman Filter to combine the data from RTK GPS and the IMU to improve the accuracy of the geo-registration and the localization result of the ground targets. The localization results are compared to the reference point coordinates from satellite image. The comparison indicates that the proposed method can provide practical geo-registration and target localization results.