LiDAR-camera fusion can enhance the performance of 3D object detection by utilizing complementary information between depth-aware LiDAR points and semantically rich images. Existing voxel-based methods face significant challenges when fusing sparse voxel features with dense image features in a one-to-one manner, resulting in the loss of the advantages of images, including semantic and continuity information, leading to sub-optimal detection performance, especially at long distances. In this paper, we present VoxelNextFusion, a multi-modal 3D object detection framework specifically designed for voxel-based methods, which effectively bridges the gap between sparse point clouds and dense images. In particular, we propose a voxel-based image pipeline that involves projecting point clouds onto images to obtain both pixel- and patch-level features. These features are then fused using a self-attention to obtain a combined representation. Moreover, to address the issue of background features present in patches, we propose a feature importance module that effectively distinguishes between foreground and background features, thus minimizing the impact of the background features. Extensive experiments were conducted on the widely used KITTI and nuScenes 3D object detection benchmarks. Notably, our VoxelNextFusion achieved around +3.20% in AP@0.7 improvement for car detection in hard level compared to the Voxel R-CNN baseline on the KITTI test dataset.
In recent years, dominant multi-object tracking (MOT) and segmentation (MOTS) methods mainly follow the tracking-by-detection paradigm. Transformer-based end-to-end (E2E) solutions bring some ideas to MOT and MOTS, but they can not achieve a new state-of-the-art (SOTA) performance in major MOT and MOTS bench-marks. Detection and association are two main modules of the tracking-by-detection paradigm. Association techniques mainly depend on the combination of motion and appearance information. As deep learning has been recently developed, the performance of the detection and appearance model is rapidly improved. These trends made us consider whether we can achieve SOTA based on only high-performance detection and appearance model. Our paper mainly focuses on exploring this direction based on CBNetV2 with Swin-B as a detection model and MoCo-v2 as a self-supervised appearance model. Motion information and IoU mapping were removed during the association. Our method achieves SOTA results on 2 mainstream MOT datasets and 1 MOTS dataset which is BDD100K MOT, WAYMO 2D Tracking, BDD100K MOTS. Our method yielded a significant improvement of +10.7% and +33.7%, respectively on BDD 100K MOT and MOTS benchmark. The proposed method won first place in BDD100K Multiple Object Tracking (MOT) challenges at CVPR 2022 Workshop on Autonomous Driving. Our method also won first place in BDD100K Multiple Object Tracking (MOT) and Multiple Object Tracking and Segmentation (MOTS) challenges at ECCV 2022 Self-supervised Learning for Next-Generation Industry-level Autonomous Driving (SSLAD) Workshop. We hope our simple and effective method can give some insights to the MOT and MOTS research community. Source code will be released under this git repository https://github.com/CarlHuangNuc/MOT_MOTS_without_motion.
25 October 2023 in IEEE Geoscience and Remote Sensing Letters
Large-size images cannot be directly put into GPU for training and need to be cropped to patches due to GPU memory limitation. The commonly used cropping methods before are random cropping and sequential cropping, which are crude and fatally inefficient. Firstly, categories of datasets are often imbalanced, and just simple cropping misses an excellent opportunity to make the data distribution balanced. Secondly, the training needs to crop a large number of patches to cover all patterns, which greatly increases the training time. This problem is of great practical hazards but is often overlooked by previous works. The optimal solution is to generate valuable patches. Valuable patches refer to the value to network training, i.e., the value of this patch for the convergence of the network, and the improvement of the accuracy. To this end, we propose a data-related patch proposal strategy to sample high valuable patches. The core idea is to score each patch according to the accuracy of each category, so as to perform balanced sampling. Compared with random cropping or sequential cropping, our method can improve the segmentation accuracy and accelerate the training vastly. Moreover, our method also shows great advantages over the loss-based balanced approaches. Experiments on Deepglobe and Potsdam show the excellent effect of our method.
Lane detection plays an important role in the autonomous driving system and attracts widespread interest in recent years. Despite the advantages of conventional work, such as segmentation-based and point-based methods, these models either practically require a large sum of anchors or only leverage the connection message of discrete points in the lane. Different from this, curve-based methods enable one to naturally learn the holistic representation of lanes based on the geometric semantics of curves and conduct end-to-end optimization conveniently, thus serving a promising direction. However, the existing curve-based methods encounter draws in: 1) modeling some complicated curves mathematically. 2) Simultaneously capturing geometric semantics within and between lane curves. 3) The endpoints of lane lines are susceptible to occlusion, a factor that is crucial for curve modeling. To tackle these issues, we revisit the curve-based methods and propose a novel model D ynamic NUR B S Net work ( DBNet ). Specifically, we introduce the NURBS curve to model lanes, enabling to theoretically fit various complicated curves and guarantee the robustness of local and global optimization. Based on this, we introduce three key modules to address lane detection challenges. Firstly, we propose Local Dynamic-Interaction (LDI) that adaptively exploits the geometric message of local at the feature level. Secondly, a Curve Fitting Enhancement (CFE) module to enhance the feature of both ends of the lane instance. Lastly, a Global Association-Sharing (GAS) module captures the global features of lanes, which can together promote semantic association and tackle the occlusion problem inside and outside the lane. Moreover, the proposed method achieves a new state-of-the-art performance on three public benchmarks and especially achieves 77.98% of the F1 score on the curve scenario in the CULane dataset.