22 February 2023 in IEEE Robotics and Automation Letters
Semantic scene understanding using thermal images has received great attention due to the advantage that thermal imaging cameras could see in challenging illumination conditions. However, thermal images are lack of color information and the edges in thermal images are often blurred, making them not very suitable to be directly used by existing semantic segmentation networks that are designed with RGB images. To address this problem, we propose a cross-modal edge-privileged knowledge distillation framework, which utilizes a well-trained RGB-Thermal fusion-based semantic segmentation network with edge-privileged information as the teacher, to guide the training of a semantic segmentation network as the student. The student network only uses thermal images. The experimental results on a public dataset demonstrate that under the guidance of the teacher, the student network achieves superior performance over the state of the arts using only thermal images.
22 August 2022 in IEEE Transactions on Instrumentation and Measurement
Road potholes can cause discomforts to passengers and even traffic accidents to vehicles. Accurate segmentation of road potholes is an important capability for autonomous vehicles to ensure safe driving. Some methods on road-pothole segmentation use single-modal data (i.e., RGB images). The main challenge faced by these methods is that the visual appearance of road potholes is often close to road areas, making these networks difficult to distinguish them. Recent methods resort to fusing RGB images with depth/disparity images for pothole segmentation. However, their performance is still not satisfactory in real-world applications. To achieve superior results, this article proposes a novel data fusion network for road-pothole segmentation, where a channel attention fusion module and a dual attention fusion (DAF) module are designed to hierarchically fuse the RGB and disparity data. We evaluate our proposed network using a public dataset, and the experimental results demonstrate the superiority over the state-of-the-art networks.
18 March 2024 in IEEE Transactions on Intelligent Vehicles
Segmentation of road negative obstacles (i.e., potholes and cracks) is important to the safety of autonomous driving. Although existing RGB-D fusion networks could achieve acceptable performance, most of them only conduct binary segmentation for negative obstacles, which does not distinguish potholes and cracks. Moreover, their performance is susceptible to depth noises, in which case the fluctuations of depth data caused by the noises may make the networks mistakenly treat the area as a negative obstacle. To provide a solution to the above issues, we design a novel RGB-D semantic segmentation network with dual semantic-feature complementary fusion for road negative obstacle segmentation. We also re-label an RGB-D dataset for this task, which distinguishes road potholes and cracks as two different classes. Experimental results show that our network achieves state-of-the-art performance compared to existing well-known networks.
2 May 2023 in IEEE Robotics and Automation Letters
Segmentation of road obstacles, including negative and positive obstacles, is critical to the safe navigation of autonomous vehicles. Recent methods have witnessed an increasing interest in using multi-modal data fusion (e.g., RGB and depth/disparity images). Although improved segmentation accuracy has been achieved by these methods, we still find that their performance could be easily degraded if the two modalities have inconsistent information, for example, distant obstacles that can be viewed in RGB images but cannot be viewed in depth/disparity images. To address this issue, we propose a novel two-encoder-two-decoder RGB-depth/disparity multi-modal network with Residual-Guided Fusion modules. Different from most existing networks that fuse feature maps in encoders, we fuse feature maps in decoder. We also release a large-scale RGB-depth/disparity dataset recorded in both urban and rural environments with manually-labeled ground truth for both negative- and positive-obstacles segmentation. Extensive experimental results demonstrate that our network achieves state-of-the-art performance compared with other networks.
7 October 2024 in IEEE Transactions on Intelligent Transportation Systems
In recent years, deep learning-based end-to-end autonomous driving has become increasingly popular. However, deep neural networks are like black boxes. Their outputs are generally not explainable, making them not reliable to be used in real-world environments. To provide a solution to this problem, we propose an explainable deep neural network that jointly predicts driving actions and multimodal environment descriptions of traffic scenes, including bird-eye-view (BEV) maps and natural-language environment descriptions. In this network, both the context information from BEV perception and the local information from semantic perception are considered before producing the driving actions and natural-language environment descriptions. To evaluate our network, we build a new dataset with hand-labelled ground truth for driving actions and multimodal environment descriptions. Experimental results show that the combination of context information and local information enhances the prediction performance of driving action and environment description, thereby improving the safety and explainability of our end-to-end autonomous driving network.
23 September 2025 in IEEE Transactions on Automation Science and Engineering
Road pothole segmentation is important for the driving safety of autonomous vehicles, especially in unstructured or rural environments. Recently, many effective multi-modal fusion networks have been proposed for road pothole segmentation. However, most of them adopt two encoders with the same type of structure, such as only convolutional neural network (CNN) or only Transformer, to extract features from different modalities. This overlooks the fact that the information richness of features extracted from different modalities varies across the types of features, such as CNN features or self-attention features. To provide a solution to this issue, we design a novel RGB-Disparity segmentation network, named MMFSeg, by adopting the two types of structures as encoders. Specifically, we adopt CNN and Transformer as encoders to extract two types of features from each modality, and also propose a late-fusion multi-feature alignment fusion module to fuse the two types of features with different numbers of channels. Experimental results demonstrate that our network outperforms well-known networks, and can trade-off between accuracy and efficiency.