14 November 2022 in IEEE Transactions on Intelligent Vehicles
Accurate and reliable object detection is a fundamental component of perception system for autonomous driving. Specially, in some circumstances like autonomous driving in surface mine, there is a fact that the particularity of scene brings tremendous challenges for object detection with a series of problems caused by the multi-scale and camouflaged objects. In this paper, a multi-scale feature fusion and attention based multi-branches framework was proposed to improve the performance of object detection for above problems called MSFANet. In the proposed MSFANet, a multi-scale feature fusion module, which was used to capture the rich context features for multi-scale high level feature maps, and a multi-scale attention module, which was used to enhance the feature saliency of objects with different scales, were designed. What's more, to improve the performance of multi-scale object detection, we build 4 different prediction branches for large, medium small and smaller scale objects respectively. At last, we built our own dataset for automatic driving in surface mine called SurMine and test the model at our own datasets and KITTI benchmark. It achieved 82.7 mAP(%) and 92.57 mAP(%) in 32 36 ms on a TITAN RTX, compared to 80.2 mAP(%) and 87.83 mAP(%) in 28 ${\sim }$ 34 ms by YOLOv7 on SurMine and KITTI benchmarks.
19 July 2023 in IEEE Transactions on Intelligent Vehicles
Efficient and accurate road boundary detection is a fundamental building component of the perception system for autonomous driving. Specially, the challenges for road boundary detection in surface mine are high generalization error of model and difficulty in boundary generation, which caused by diversity of samples along with scarcity for corresponding samples and complexity of terrain respectively. Therefore, a novel road boundary detection framework, which execute in a high efficiency way with considerable performance, is proposed for the problems mentioned above. Firstly, point cloud pre-processing methods, including point cloud down-sampling, filtering and clustering, are conducted for achieving clusters of objects in surface mine. Then, a meta random forest classification method, which combines meta learning and random forest for enhancing the generalization ability of the model and overcoming sample scarcity of surface mine, is proposed for classifying point cloud clusters of retaining wall on both side of the road. At last, the boundary of unstructured road is generated by conducting a series of post-processing methods corresponds to the unevenness and irregularity of unstructured road. Experiments are carried out on the collected and labeled datasets of surface mine. The results illustrate that our proposed method can effectively detect road boundary of surface mine in real-time with considerable performance.
14 December 2023 in IEEE Transactions on Intelligent Vehicles
LiDAR-Camera fusion can effectively provide the complementary geometric and appearance information for 3D object detection task of autonomous driving system. However, current dominant methods designed for urban scenes often fuse multi-modal information into a Bird's-Eye-View (BEV) feature representation and have a relatively short perception range, which limits their application in surface mine. To achieve long-range perception in surface mine, we utilize a solid-state LiDAR (i.e. Livox Horizon) that has a large perception range (over 200 meters). Accordingly, we propose PVFusion, a novel 3D object detection method that only fuses LiDAR and camera information in Perspective View (PV). Specifically, we first generate the multi-channel depth map via point cloud projection and Depth Map Densification (DMD). Then, we extract visual-depth features and local geometry features from depth-enhanced camera image and point cloud respectively. To better fuse multi-modal information, we propose an Attentional Feature Fusion (AFF) module to dynamically fuse visual-depth features and local geometry features into fused PV features. Moreover, we design a three-branch detection head that leverages a Depth Decoupling Strategy (DDS) to improve the depth estimation for distant objects. We conduct extensive experiments on our surface mine dataset and verify the effectiveness of the proposed PVFusion.
1 October 2024 in IEEE Intelligent Transportation Systems Magazine
Synthetic data, complementary with real-world datasets, have shown their effectiveness for autonomous driving in urban scenarios. However, for specific and complex environments, such as surface mines, which often involve extreme weather and corner cases, synthetic datasets are notably scarce. In this article, PMScenes, the first synthetic dataset for autonomous driving in surface mines, is built upon the previously public AutoMine dataset. It extends the available data by constructing corner case scenarios, such as unusual intrusions; collisions; and extreme weather scenarios including rain, blizzards, fog, and dust storms—situations that would bring significant safety risks and are hard to capture in the real world. Our article explores the enhancement methods for synthetic to real for perception tasks of autonomous driving under different challenging scenarios. The outcomes demonstrate that PMScenes significantly enhances the performance across a variety of perception tasks compared with the results obtained when training only with real-world datasets. PMScenes is released at https://github.com/PMScenes/PMScenes .