Event cameras (EC) represent a paradigm shift and are emerging as valuable tools in the automotive industry, particularly for in-cabin and out-of-cabin monitoring. These cameras capture pixel intensity changes as ”events” with ultra-low latency, making them suitable for real- time applications. In the context of in-cabin monitoring, EC offer solution for driver and passenger tracking, enhancing safety and comfort. For out-of-cabin monitoring, they excel in tracking objects and detecting potential hazards on the road. This article explores the applications, benefits, and challenges of event cameras in these two critical domains within the automotive industry. This review also highlights relevant datasets and methodologies, enabling researchers to make informed decisions tailored to their specific vehicular-technology and place their work in the broader context of EC sensing. Through an exploration of the hardware, the complexities of data processing, and customized algorithms for both in-cabin and out-of-cabin surveillance, this paper outlines a framework encompassing methodologies, tools, and datasets critical for the implementation of event camera sensing in automotive systems.

AI-based smart thermal perception systems can cater to the limitations of conventional imaging sensors by providing a more reliable data source in low-lighting conditions and adverse weather conditions. This research evaluates and modifies the state-of-the-art object detection and classifier framework for thermal vision with seven key object classes in order to provide superior thermal sensing and scene understanding input for advanced driver-assistance systems (ADAS). The networks are trained on public datasets and is validated on test data with three different test approaches which include test-time augmentation, test-time with no augmentation, and test-time with model ensembling. Additionally, a new model ensemble-based inference engine is proposed, and its efficacy is tested on locally gathered novel test data comprising of 20K thermal frames captured with an uncooled LWIR prototype thermal camera in challenging weather and environmental scenarios. The performance analysis of trained models is investigated by computing precision, recall, and mean average precision scores (mAP). Furthermore, the smaller network variant of thermal-YOLO architecture is optimized using TensorRT inference accelerator, which is then deployed on GPU and resource-constrained edge hardware Nvidia Jetson Nano. This is implemented to explicitly reduce the inference time on GPU as well as on Nvidia Jetson Nano to evaluate the feasibility for added real-time onboard installations.

For decades, the number of automobiles in urban areas around the world has been increasing. It causes serious challenges such as traffic congestion, accidents, and pollution, which have a social, economic, and environmental impact on widespread urban cities. To overcome these challenges, we need to explore smart AI-based perception systems for vehicular applications. Such types of systems can provide improved situational awareness to the driver and generate early alarm about upcoming obstacles and road incidents. In this study, we have presented the effective use of uncooled thermal IR sensors for designing smart thermal perception systems as an alternative to CMOS visible imaging by presenting state-of-the-art studies for in-cabin and out-cabin vehicular applications with potential long-term benefits for the automotive industry. The key rationale for selecting thermal IR sensors over conventional image sensors is that visible cameras are highly dependent on lighting conditions and performance is degraded significantly in low-lighting scenarios and harsh weather conditions. Contrary to this, thermal sensors remain largely unaffected by external lighting conditions or most environmental conditions, making them a perfect optical sensor choice for all-weather and harsh environmental conditions. This study presents a review of the current state of the art for automotive thermal imaging with a focus on the contributions and advances achieved by the EU-funded project ‘HELIAUS’ in the domain of AI-based thermal imaging pipelines for safer and reliable road journeys.

Event cameras, unlike traditional frame-based cameras, excel in detecting and reporting changes in light intensity on a per-pixel basis. This unique technology offers numerous advantages, including high temporal resolution, low latency, wide dynamic range, and reduced power consumption. These characteristics make event cameras particularly well-suited for sensing applications such as monitoring drivers or human behavior. This paper presents a feasibility study on the using a multitask neural network with event cameras for real-time facial analytics. Our proposed network simultaneously estimates head pose, eye gaze, and facial occlusions. Notably, the network is trained on synthetic event camera data, and its performance is demonstrated and validated using real event data in real-time driving scenarios. To compensate for global head motion, we introduce a novel event integration method capable of handling both short and long-term temporal dependencies. The performance of our facial analytics method is quantitatively evaluated in both controlled lab environments and unconstrained driving scenarios. The results demonstrate that useful accuracy and computational speed is achieved by the proposed method to determining head pose and relative eye-gaze direction. This shows that neuromorphic facial analytics can be realized in real-time and are well-suited for edge/embedded computing deployments. While the improvement ratio in comparison to existing literature may not be as favorable due to the unique event-based vision approach employed, it is crucial to note that our research focuses specifically on event-based vision, which offers distinct advantages over traditional RGB vision. Overall, this study contributes to the emerging field of event-based vision systems and highlights the potential of multitask neural networks combined with event cameras for real-time sensing of human subjects. These techniques can be applied in practical applications such as driver monitoring systems, interactive human-computer systems and for human behavior analysis.

Neuromorphic vision sensors, commonly referred to as Event Cameras (ECs), have gained prominence as a field of research in Computer Vision. This popularity stems from the numerous unique characteristics including High Dynamic Range, High Temporal Resolution, and Low Latency. Of particular interest is their temporal resolution, which proves ideal for human monitoring applications. Capturing rapid facial movements and eye gaze can be effectively achieved with ECs. Recent studies involving the use of ECs for object detection and tracking have demonstrated success in tasks involving Eye Motion Analysis such as Eye tracking, Blink detection, Gaze estimation and Pupil tracking. The objective of this study is to provide a comprehensive review of the current research in the aforementioned tasks, focusing on the potential utilization of ECs for future tasks involving rapid eye motion detection, such as detection and classification of saccades. We highlight studies that may serve as a foundation for undertaking such a task, such as pupil tracking and gaze estimation. We also highlight in our review some common challenges encountered such as the availability of datasets and review some of the methods used in solving this problem. Finally, we discuss some limitations of this field of research and conclude with future directions including real-world applications and potential research directions.

This study is focused on evaluating the real-time performance of thermal object detection for smart and safe vehicular systems by deploying the trained networks on GPU & single-board EDGE-GPU computing platforms for onboard automotive sensor suite testing. A novel large-scale thermal dataset comprising of > 35,000 distinct frames is acquired, processed, and open-sourced in challenging weather and environmental scenarios. The dataset is a recorded from lost-cost yet effective uncooled LWIR thermal camera, mounted stand-alone and on an electric vehicle to minimize mechanical vibrations. State-of-the-art YOLO-V5 networks variants are trained using four different public datasets as well newly acquired local dataset for optimal generalization of DNN by employing SGD optimizer. The effectiveness of trained networks is validated on extensive test data using various quantitative metrics which include precision, recall curve, mean average precision, and frames per second. The smaller network variant of YOLO is further optimized using TensorRT inference accelerator to explicitly boost the frames per second rate. Optimized network engine increases the frames per second rate by 3.5 times when testing on low power edge devices thus achieving 11 fps on Nvidia Jetson Nano and 60 fps on Nvidia Xavier NX development boards.