8 May 2024 in IEEE Communications Surveys & Tutorials
Millimeter-wave (mmWave) radar, with its high resolution, sensitivity to micro-vibrations, and adaptability to various environmental conditions, holds immense potential across multi-modal fusion sensing. Although there exist review papers on mmWave radar, there is a noticeable lack of comprehensive reviews focusing on its multi-modal fusion sensing capabilities. Addressing this gap, our review offers an extensive exploration of mmWave radar multi-modal fusion sensing, emphasizing its integration with other modalities. This review discusses the complex realm of millimeter-wave radar multi-modal fusion sensing, detailing its importance, hardware and software aspects, principles, and current potential and applications. It delves into data characteristics and datasets associated with mmWave radar, focusing on Doppler, point cloud, and multi-modal data formats. The review highlights how these data types enhance multi-modal fusion sensing and discusses methodologies, including signal processing and learning algorithms. Three categories of multi-modal fusion methodologies are proposed to optimally manage and interpret fused data. Various practical applications of mmWave radar multi-modal fusion sensing are illustrated, underlining the unique capabilities it provides when integrated with other sensors. The review concludes by identifying potential future research avenues, underscoring the immense potential of this field for further exploration and advancement.
27 March 2023 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Human sensing and understanding is a key requirement for many intelligent systems, such as smart monitoring, human-computer interaction, and activity analysis, etc. In this paper, we present mmParse, the first human parsing design for dynamic point cloud from commercial millimeter-wave radar devices. mmParse proposes an end-to-end neural network design that addresses the inherent challenges in parsing mmWave point cloud (e.g., sparsity and specular reflection). First, we design a novel multi-task learning approach, in which an auxiliary task can guide the network to understand human structural features. Secondly, we introduce a multi-task feature fusion method that incorporates both intra-task and inter-task attention to aggregate spatio-temporal features of the subject from a global view. Through extensive experiments in both indoor and outdoor environments, we demonstrate that our proposed system is able to achieve ~ 92% accuracy and ~ 84% IoU accuracy. We also show that the predicted semantic labels can increase the performance of two downstream tasks (pose estimation and action recognition) by ~ 18% and ~ 6% respectively.
3 March 2025 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Wireless localization, especially using ubiquitous Wi-Fi devices, has been studied for decades. However, due to complex signal propagation, resolving multiple reflected paths and further achieving accurate localization face great challenges in the localization precision and localization range, which are caused by the limited bandwidth and sensitivity of Wi-Fi signals respectively. Straightforwardly solving these challenges by employing wide bandwidth Wi-Fi is only helpful for improving precision, while the localization range is still short due to Wi-Fi's intrinsic vulnerability to SNR drop. This paper presents Wi-Fi-UWB Localization (WULoc) to achieve both precise and long-range localization for Wi-Fi devices. Specifically, WULoc leverages the picosecond level timestamp information that is extracted from the links of the Wi-Fi device to multiple UWB anchors, innovatively created by the proposed Wi-Fi-UWB connection establishment, which overcomes the incompatibility between Wi-Fi and UWB physical layers. WULoc further calculates the TDoA of Wi-Fi-UWB links, from which the location of the Wi-Fi device is estimated. Extensive experiments show that WULoc locates a Wi-Fi device at a range of 240 meters with a precision of 5.3 meters.