In this paper, mm-Pose, a novel approach to detect and track human skeletons in real-time using an mmWave radar, is proposed. To the best of the authors’ knowledge, this is the first method to detect >15 distinct skeletal joints using mmWave radar reflection signals. The proposed method would find several applications in traffic monitoring systems, autonomous vehicles, patient monitoring systems and defense forces to detect and track human skeleton for effective and preventive decision making in real-time. The use of radar makes the system operationally robust to scene lighting and adverse weather conditions. The reflected radar point cloud in range, azimuth and elevation are first resolved and projected in Range-Azimuth and Range-Elevation planes. A novel low-size high-resolution radar-to-image representation is also presented, that overcomes the sparsity in traditional point cloud data and offers significant reduction in the subsequent machine learning architecture. The RGB channels were assigned with the normalized values of range, elevation/azimuth and the power level of the reflection signals for each of the points. A forked CNN architecture was used to predict the real-world position of the skeletal joints in 3-D space, using the radar-to-image representation. The proposed method was tested for a single human scenario for four primary motions, (i) Walking, (ii) Swinging left arm, (iii) Swinging right arm, and (iv) Swinging both arms to validate accurate predictions for motion in range, azimuth and elevation. The detailed methodology, implementation, challenges, and validation results are presented.
A real-time motion behavior detection system using millimeter wave (mmWave) radar is presented in this paper. Radar is used to sense the micro-Doppler information of targets. A convolution neural network (CNN) is further implemented in detecting and classifying the human behaviors using this information. Both the convolution layers and architecture of CNNs are presented. The analysis on loss and accuracy of training results are also shown. The experimental result indicates a precise determination of human motion behavior detection using the proposed system.
25 February 2019 in IEEE Transactions on Vehicular Technology
Interference among frequency modulated continues wave (FMCW) automotive radars can either increase the noise floor, which occurs in the most cases, or generate a ghost target in rare situations. To address the increment of noise floor due to interference, we proposed a low calculation cost method using adaptive noise canceller to increase the signal-to-interference ratio (SIR). In a quadrature receiver, the interference in the positive half of frequency spectrum is correlated to that in the negative half of frequency spectrum, while the beat frequencies from real targets are always present in the positive frequency. Thus, we estimated the power of the negative frequency as an indication of interference, and fed the positive frequency and negative frequency components into the primary and reference channel of an adaptive noise canceller, respectively. The least mean square (LMS) algorithm was used to solve for the optimum filter solution. As a result, both the simulation and experiment showed a good interference mitigation performance.
23 December 2020 in IEEE Transactions on Automation Science and Engineering
Elderly fall prevention and detection becomes extremely crucial with the fast aging population globally. In this article, we propose mmFall, a novel fall detection system, which comprises 1) the emerging millimeter-wave (mmWave) radar sensor to collect the human body's point cloud along with the body centroid and 2) a hybrid variational recurrent neural network (RNN) autoencoder (HVRAE) to compute the anomaly level of the body motion based on the acquired point cloud. A fall is detected when the spike in anomaly level and the drop in centroid height occur simultaneously. The mmWave radar sensor offers privacy-compliance and high sensitivity to motion, over the traditional sensing modalities. However, 1) randomness in radar point cloud and 2) difficulties in fall collection/labeling in the traditional supervised fall detection approaches are the two major challenges. To overcome the randomness in radar data, the proposed HVRAE uses variational inference, a generative approach rather than a discriminative approach, to infer the posterior probability of the body's latent motion state every frame, followed by a RNN to summarize the temporal features over multiple frames. Moreover, to circumvent the difficulties in fall data collection/labeling, the HVRAE is built upon an autoencoder architecture in a semisupervised approach, which is only trained on the normal activities of daily living (ADL). In the inference stage, the HVRAE will generate a spike in the anomaly level once an abnormal motion, such as fall, occurs. During the experiment,
1 March 2022 in IEEE Transactions on Neural Networks and Learning Systems
In this article, we presented mmPose-NLP, a novel natural language processing (NLP) inspired sequence-to-sequence (Seq2Seq) skeletal key-point estimator using millimeter-wave (mmWave) radar data. To the best of our knowledge, this is the first method to precisely estimate up to 25 skeletal key points using mmWave radar data alone. Skeletal pose estimation is critical in several applications ranging from autonomous vehicles, traffic monitoring, patient monitoring, and gait analysis, to defense security forensics, and aid both preventative and actionable decision making. The use of mmWave radars for this task, over traditionally employed optical sensors, provides several advantages, primarily its operational robustness to scene lighting and adverse weather conditions, where optical sensor performance degrade significantly. The mmWave radar point-cloud (PCL) data are first voxelized (analogous to tokenization in NLP) and $N$ frames of the voxelized radar data (analogous to a text paragraph in NLP) is subjected to the proposed mmPose-NLP architecture, where the voxel indices of the 25 skeletal key points (analogous to keyword extraction in NLP) are predicted. The voxel indices are converted back to real-world 3-D coordinates using the voxel dictionary used during the tokenization process. Mean absolute error (MAE) metrics were used to measure the accuracy of the proposed system against the ground truth, with the proposed mmPose-NLP offering < 3 cm localization errors in the depth, horizontal, and vertical axes. The effect of the number of input frames versus performance/accuracy was also studied for $N = \{1,2, {\dots },10\}$ . A comprehensive methodology, results, discussions, and limitations are presented in this article. All the source codes and results are made available on GitHub for further research and development in this critical yet emerging domain of skeletal key-point estimation using mmWave radars.
To address potential gaps noted in patient monitoring in the hospital, a novel patient behavior detection system using mmWave radar and deep convolution neural network (CNN), which supports the simultaneous recognition of multiple patients' behaviors in real-time, is proposed. In this study, we use an mmWave radar to track multiple patients and detect the scattering point cloud of each one. For each patient, the Doppler pattern of the point cloud over a time period is collected as the behavior signature. A three-layer CNN model is created to classify the behavior for each patient. The tracking and point clouds detection algorithm was also implemented on an mmWave radar hardware platform with an embedded graphics processing unit (GPU) board to collect Doppler pattern and run the CNN model. A training dataset of six types of behavior were collected, over a long duration, to train the model using Adam optimizer with an objective to minimize cross-entropy loss function. Lastly, the system was tested for real-time operation and obtained a very good inference accuracy when predicting each patient's behavior in a two-patient scenario.