LiDAR is a commonly used sensor for autonomous driving to make accurate, robust, and fast decision-making when driving. The sensor is used in the perception system, especially object detection, to understand the driving environment. Although 2D object detection has succeeded during the deep-learning era, the lack of depth information limits understanding of the driving environment and object location. Three-dimensional sensors, such as LiDAR, give 3D information about the surrounding environment, which is essential for a 3D perception system. Despite the attention of the computer vision community to 3D object detection due to multiple applications in robotics and autonomous driving, there are challenges, such as scale change, sparsity, uneven distribution of LiDAR data, and occlusions. Different representations of LiDAR data and methods to minimize the effect of the sparsity of LiDAR data have been proposed. This survey presents the LiDAR-based 3D object detection and feature-extraction techniques for LiDAR data. The 3D coordinate systems differ in camera and LiDAR-based datasets and methods. Therefore, the commonly used 3D coordinate systems are summarized. Then, state-of-the-art LiDAR-based 3D object-detection methods are reviewed with a selected comparison among methods.

An accurate and robust perception system is key to understanding the driving environment of autonomous driving and robots. Autonomous driving needs 3D information about objects, including the object’s location and pose, to understand the driving environment clearly. A camera sensor is widely used in autonomous driving because of its richness in color and texture and low price. The major problem with the camera is the lack of 3D information, which is necessary to understand the 3D driving environment. Additionally, the object’s scale change and occlusion make 3D object detection more challenging. Many deep learning-based methods, such as depth estimation, have been developed to solve the lack of 3D information. This survey presents the image 3D object detection 3D bounding box encoding techniques and evaluation metrics. The image-based methods are categorized based on the technique used to estimate an image’s depth information, and insights are added to each method. Then, state-of-the-art (SOTA) monocular and stereo camera-based methods are summarized. We also compare the performance of the selected 3D object detection models and present challenges and future directions in 3D object detection.

The pursuit of autonomous driving relies on developing perception systems capable of making accurate, robust, and rapid decisions to interpret the driving environment effectively. Object detection is crucial for understanding the environment at these systems’ core. While 2D object detection and classification have advanced significantly with the advent of deep learning (DL) in computer vision (CV) applications, they fall short in providing essential depth information, a key element in comprehending driving environments. Consequently, 3D object detection becomes a cornerstone for autonomous driving and robotics, offering precise estimations of object locations and enhancing environmental comprehension. The CV community’s growing interest in 3D object detection is fueled by the evolution of DL models, including Convolutional Neural Networks (CNNs) and Transformer networks. Despite these advancements, challenges such as varying object scales, limited 3D sensor data, and occlusions persist in 3D object detection. To address these challenges, researchers are exploring multimodal techniques that combine information from multiple sensors, such as cameras, radar, and LiDAR, to enhance the performance of perception systems. This survey provides an exhaustive review of multimodal fusion-based 3D object detection methods, focusing on CNN and Transformer-based models. It underscores the necessity of equipping fully autonomous vehicles with diverse sensors to ensure robust and reliable operation. The survey explores the advantages and drawbacks of cameras, LiDAR, and radar sensors. Additionally, it summarizes autonomy datasets and examines the latest advancements in multimodal fusion-based methods. The survey concludes by highlighting the ongoing challenges, open issues, and potential directions for future research.

Fish species recognition is crucial to identifying the abundance of fish species in a specific area, controlling production management, and monitoring the ecosystem, especially identifying the endangered species, which makes accurate fish species recognition essential. In this work, the fish species recognition problem is formulated as an object detection model to handle multiple fish in a single image, which is challenging to classify using a simple classification network. The proposed model consists of MobileNetv3-large and VGG16 backbone networks and an SSD detection head. Moreover, a class-aware loss function is proposed to solve the class imbalance problem of our dataset. The class-aware loss takes the number of instances in each species into account and gives more weight to those species with a smaller number of instances. This loss function can be applied to any classification or object detection task with an imbalanced dataset. The experimental result on the large-scale reef fish dataset, SEAMAPD21, shows that the class-aware loss improves the model over the original loss by up to 79.7%. The experimental result on the Pascal VOC dataset also shows the model outperforms the original SSD object detection model.

Three-dimensional object detection is crucial for autonomous driving to understand the driving environment. Since the pooling operation causes information loss in the standard CNN, we designed a wavelet-multiresolution-analysis-based 3D object detection network without a pooling operation. Additionally, instead of using a single filter like the standard convolution, we used the lower-frequency and higher-frequency coefficients as a filter. These filters capture more relevant parts than a single filter, enlarging the receptive field. The model comprises a discrete wavelet transform (DWT) and an inverse wavelet transform (IWT) with skip connections to encourage feature reuse for contrasting and expanding layers. The IWT enriches the feature representation by fully recovering the lost details during the downsampling operation. Element-wise summation was used for the skip connections to decrease the computational burden. We trained the model for the Haar and Daubechies (Db4) wavelets. The two-level wavelet decomposition result shows that we can build a lightweight model without losing significant performance. The experimental results on KITTI’s BEV and 3D evaluation benchmark show that our model outperforms the PointPillars-based model by up to 14% while reducing the number of trainable parameters.

In video-based fish surveys, species recognition plays a vital role in stock assessments, ecosystem analysis, production management, and protection of endangered species. However, implementing fish species detection algorithms in underwater environments presents significant challenges due to factors such as varying lighting conditions, water turbidity, and the diverse appearances of fish species. In this work, a transformer-enhanced YOLOv8 (YOLOv8-TF) is proposed for underwater fish species recognition. The YOLOv8-TF enhances the performance of YOLOv8 by adjusting depth scales, incorporating a transformer block into the backbone and neck, and introducing a class-aware loss function to address class imbalance in the dataset. The class-aware loss considers the count of instances within each species and assigns a higher weight to species with fewer instances. This approach enables fish species recognition through object detection, encompassing the classification of each fish species and localization to estimate their position and size within an image. Experiments were conducted using the 2021 Southeast Area Monitoring and Assessment Program (SEAMAPD21) dataset, a detailed and extensive reef fish dataset from the Gulf of Mexico. The experimental results on SEAMAPD21 demonstrate that the YOLOv8-TF model, with a mean Average Precision (mAP)_0.5 of 87.9% and mAP_0.5–0.95 of 61.2%, achieves better detection results for underwater fish species recognition compared to state-of-the-art YOLO models. Additionally, experimental results on the publicly available datasets, such as Pascal VOC and MS COCO datasets demonstrate that the model outperforms existing approaches.