A pedestrian detection system is a crucial component of advanced driver assistance systems since it contributes to road flow safety. The safety of traffic participants could be significantly improved if these systems could also predict and recognize pedestrian’s actions, or even estimate the time, for each pedestrian, to cross the street. In this paper, we focus not only on pedestrian detection and pedestrian action recognition but also on estimating if the pedestrian’s action presents a risky situation according to time to cross the street. We propose 1) a pedestrian detection and action recognition component based, on RetinaNet; 2) an estimation of the time to cross the street for multiple pedestrians using a recurrent neural network. For each pedestrian, the recurrent network estimates the pedestrian’s action intention in order to predict the time to cross the street. We based our experiments on the JAAD dataset, and show that integrating multiple pedestrian action tags for the detection part when merge with a recurrent neural network (LSTM) allows a significant performance improvement.
7 February 2020 in IEEE Intelligent Transportation Systems Magazine
The combination of multi-modal image fusion schemes with deep learning classification methods, and particularly with Convolutional Neural Networks (CNNs) has achieved remarkable performances in the pedestrian detection field. The late fusion scheme has significantly enhanced the performance of the pedestrian recognition task. In this paper, the late fusion scheme connected with CNN learning is deeply investigated for pedestrian recognition based on the Daimler stereo vision dataset. Thus, an independent CNN for each imaging modality (Intensity, Depth, and Optical Flow) is used before the fusion of the CNN?s probabilistic output scores with a Multi-Layer Perceptron which provides the recognition decision. We propose four different learning patterns based on Cross-Modality deep learning of Convolutional Neural Networks: (1) a Particular Cross-Modality Learning; (2) a Separate Cross-Modality Learning; (3) a Correlated Cross-Modality Learning and (4) an Incremental Cross-Modality Learning model. Moreover, we also design a new CNN architecture, called LeNet+, which improves the classification performance not only for each modality classifier, but also for the multi-modality late-fusion scheme. Finally, we propose to learn the LeNet+ model with the incremental cross-modality approach using optimal learning settings, obtained with a K-fold Cross Validation pattern. This method outperforms the state-of-the-art classifier provided with Daimler datasets on both non-occluded and partially-occluded pedestrian tasks.
Detecting pedestrians at high speed in high resolution (HR) images is important for preventing collisions in autonomous vehicle. The HR images provide more detail information while creating a heavy computational load, resulting in slower detection speed. We find that the pedestrians are usually not present in the upper part of the HR image, due to the camera vertical field of view and the distance between pedestrian and camera. We named this area a non-pedestrian area (NPA). Therefore, we are able to reduce the calculation time while maintaining the detection accuracy by removing NPA from the HR image. According to this concept, we propose a novel pedestrian detection acceleration algorithm called Non-Pedestrian Area Estimation (NPAE). The NPAE algorithm estimates and removes non-pedestrian areas of the image, followed by pedestrian detection of the NPAE output image. Two datasets with three different resolution images are used to test our proposed NPAE algorithm on both GPU and CPU platforms. The RetinaNet is chosen as the reference pedestrian detector. Our algorithm increases the detection speed with 42% on the GPU platform and with 52% speed increase on the CPU platform, for the test images with a resolution of $1920 \times 1080$ pixel. In terms of detection accuracy, NPAE improves by up to 37.94% compared to classical data optimization acceleration methods. The detection accuracy results are well preserved in both platforms.
Pedestrian detection is a fundamental component of Advanced Driver Assistance Systems (ADAS) and plays a key role in collision avoidance and the safety of vulnerable road users. This paper presents a structured review of pedestrian detection methodologies developed between 2000 and 2025, spanning classical vision techniques and modern deep learning architectures. We organize the review into two phases. First, we examine classical methods, including Histogram of Oriented Gradients (HOG)+Support Vector Machine (SVM), Viola–Jones, Deformable Part Models, and Integral Channel Features, which established the conceptual foundations of the field. Then, we analyze state-of-the-art deep learning architectures, categorized by detector stage (one-stage vs. two-stage), localization strategy (anchor-based vs. anchor-free), feature extraction paradigm (Convolutional Neural Network (CNN)-based vs. transformer-based), output representation (bounding box vs. instance segmentation), and computational profile (lightweight vs. heavyweight). Several design principles introduced by classical methods remain visible in modern architectures, indicating that they were not fully superseded. The review also examines publicly available benchmark datasets and compares the strengths and limitations of camera-, Light Detection And Ranging (LiDAR)-, radar-, and multi-sensor-fusion-based systems for ADAS deployment. We close by identifying six open problems for the field: adversarial robustness, real-time inference under embedded constraints, detection under adverse weather, dataset bias and demographic fairness, the deployment of Bird’s-Eye View (BEV) and unified perception on automotive hardware, and explainability for safety-critical use.