Automotive perception involves understanding the external driving environment as well as the internal state of the vehicle cabin and occupants using sensor data. It is critical to achieving high levels of safety and autonomy in driving. This paper provides an overview of different sensor modalities like cameras, radars, and LiDARs used commonly for perception, along with the associated data processing techniques. Critical aspects in perception are considered, like architectures for processing data from single or multiple sensor modalities, sensor data processing algorithms and the role of machine learning techniques, methodologies for validating the performance of perception systems, and safety. The technical challenges for each aspect are analyzed, emphasizing machine learning approaches given their potential impact on improving perception. Finally, future research opportunities in automotive perception for their wider deployment are outlined.

Gesture Recognition is one of the most intuitive form of human computer interface. Gesture sensing can replace interfaces such as touch and clicks needed for interacting with a device. In this letter, we present a short-range compact 60- GHz mmwave radar sensor that is sensitive to fine dynamic hand motions. A series of range-Doppler images are extracted and processed using a Long Recurrent All-Convolution Neural Network (LRACN) for real-time dynamic hand gesture recognition. Furthermore, we make use of novel data augmentation techniques for the proposed gesture recognition system to generalize for multiple users and operating environments. The results show accurate classification performance requiring very low processor footprint facilitating implementation in embedded platforms with real time user-feedback.

This paper presents a millimeter wave radar in the 24-GHz ISM band for detection, tracking, and classification of human targets. Linear frequency modulation of the transmit signal and two receive antennas enable distance and angle measurements, respectively. Multiple consecutive frequency chirps are used for target velocity calculation. Hardware as well as firmware concepts of the proposed system are described in detail. Various algorithms for human detection and tracking are investigated and combined to a new signal processing routine optimized for compactness and low-power to run on a microcontroller. Additionally, a novel Doppler-compensated angle-ofarrival estimation method as well as a one-class support vector machine for human classification are proposed to further enhance the human detection and tracking performance. The achieved performances of the designed hardware and the implemented algorithm are verified in extensive measurements. The distance and angle errors of the realized radar sensor are at most 25 cm along a measurement range of 18m and 10° for a two-sided angle sweep of 65°, respectively. The achieved range resolution is 0.9 m. Dedicated verifications of the most important signal processing routines are presented to verify their functionality and experiments with several human targets illustrate the performance and limits of the overall tracking algorithm. It is shown that range, velocity, and angle of up to five humans are correctly detected and tracked. The presented one-class classifier successfully distinguishes human targets from other quasi-static targets like trees and shadowing effects of human subjects on walls.

Radar systems enable a contactless, non-invasive method to simultaneously monitor the respiration and heart rate of a human target. To enhance the stability and accuracy of the detected heart rate, this paper introduces a Kalman filter-based tracking of the heartbeat signal. After providing a first rough estimation, the bandwidth of the applied band-pass filter is successively narrowed down and the filter limits are steadily updated with respect to the current heart rate of the target. Measurement segments with random body movements are automatically identified and consequentially ignored for the Kalman filter update. The functionality of the proposed algorithm is proven by various verification measurements using a 60-GHz frequency-modulated continuous-wave radar system. The determined heart rates are compared to the output of a commercial chest belt. Measurement data with common breathing as well as during breath-holding were recorded for fourteen different subjects.

Radar technology plays a vital role in contact-less detection of hand gestures or motions, which forms an alternate and intuitive form of human-computer interface. Air-writing refers to the writing of linguistic characters or words in free space by hand gesture movements. In this paper, we propose an air-writing system based on a network of milli-meter wave radars. We propose a two-stage approach for extraction and recognition of handwriting gestures. The extraction processing stage uses a fine range estimate combined with the trilateration technique to detect and localize the hand marker, followed by a smoothening filter to create a trajectory of the character through the hand movement. For the recognition stage, we explore two approaches, one extracts the time-series trajectory data and recognizes the drawn character using Long Short Term Memory (LSTM), Bi-directional LSTM (BLSTM) and Convolutional LSTM (ConvLSTM) with Connectionist Temporal Classification (CTC) loss function. The other approach reconstructs a 2D image from the trajectory of drawn character and uses Deep Convolutional Neural Network (DCNN) to classify the alphabets drawn by the user. ConvLSTM-CTC performs best among LSTM variants on time-series trajectory data achieving 98.33% classification accuracy similar to DCNN over the chosen character set. The paper employs real data using a network of three 60-GHz milli-meter wave radar sensor to demonstrate the success of the proposed setup and associated algorithm with design consideration.

Gesture recognition is the most intuitive form of human computer-interface. Gesture sensing can replace interfaces such as touch and clicks needed for interacting with a device. Gesture recognition in a practical scenario is an open-set classification, i.e. the recognition system should classify correct known gestures while rejecting arbitrary unknown gestures during inference. To address the issue of gesture recognition in an open set, we present, in this paper, a novel distance-metric based meta-learning approach to learn embedding features from a video of range-Doppler images generated by hand gestures at the radar receiver. Further, k-Nearest Neighbor (kNN) is used to classify known gestures, distance-thresholding is used to reject unknown gesture motions and clustering is used to add new custom gestures on-the-fly without explicit model re-training. We propose to use 3D Deep Convolutional Neural Network (3D-DCNN) architecture to learn the embedding model using distance-based triplet-loss similarity metric. We demonstrate our approach to correctly classify gestures using short-range 60-GHz compact short-range radar sensor achieving an overall classification accuracy of 94.5% over six fine-grained gestures under challenging practical environments, while rejecting other unknown gestures with 0.935 F1 score, and capable of adding new gestures on-the-fly without an explicit model re-training.