28 November 2023 in IEEE Communications Surveys & Tutorials
In the era of sixth-generation (6G) wireless communications, integrated sensing and communications (ISAC) is recognized as a promising solution to upgrade the physical system by endowing wireless communications with sensing capability. Existing ISAC is mainly oriented to static scenarios with radio-frequency (RF) sensors being the primary participants, thus lacking a comprehensive environment feature characterization and facing a severe performance bottleneck in dynamic environments. To date, extensive surveys on ISAC have been conducted but are limited to summarizing RF-based radar sensing. Currently, some research efforts have been devoted to exploring multi-modal sensing-communication integration but still lack a comprehensive review. To fill the gap, we embark on an initial endeavor with the goal of establishing a unified framework of intelligent multi-modal sensing-communication integration by generalizing the concept of ISAC and providing a comprehensive review under this framework. Inspired by the human synesthesia, the so-termed Synesthesia of Machines (SoM) gives the clearest cognition of such an intelligent integration and details its paradigm for the first time. We commence by justifying the necessity and potential of the new paradigm. Subsequently, we offer a rigorous definition of SoM and zoom into the detailed paradigm, which is summarized as three operational modes realizing the integration. To facilitate SoM research, we overview the prerequisite of SoM research, that is, mixed multi-modal (MMM) datasets, and introduce our work. Built upon the MMM datasets, we introduce the mapping relationships between multi-modal sensing and communications, and discuss how channel modeling can be customized to support the exploration of such relationships. Afterward, aiming at giving a comprehensive survey on the current research status of multi-modal sensing-communication integration, we cover the technological review on SoM-enhance-based and SoM-concert-based applications in transceiver design and environment sensing. To corroborate the rationality and superiority of SoM, we also present simulation results related to dual-function waveform and predictive beamforming design tailored for dynamic scenarios. Finally, we propose some open issues and potential directions to inspire future research efforts on SoM.
13 May 2024 in IEEE Robotics and Automation Letters
We introduce an integrated precise LiDAR, Inertial, and Visual (LIV) multimodal sensor fused mapping system that builds on the differentiable Gaussians to improve the mapping fidelity, quality, and structural accuracy. Notably, this is also a novel form of tightly coupled map for LiDAR-visual-inertial sensor fusion. This system leverages the complementary characteristics of LiDAR and visual data to capture the geometric structures of large-scale 3D scenes and restore their visual surface information with high fidelity. The initialization for the scene's surface Gaussians and the sensor's poses of each frame are obtained using a LiDAR-inertial system with the feature of size-adaptive voxels. Then, we optimized and refined the Gaussians using visual-derived photometric gradients to optimize their quality and density. Our method is compatible with various types of LiDAR, including solid-state and mechanical LiDAR, supporting both repetitive and non-repetitive scanning modes. Bolstering structure construction through LiDAR and facilitating real-time generation of photorealistic renderings across diverse LIV datasets. It showcases notable resilience and versatility in generating real-time photorealistic scenes potentially for digital twins and virtual reality, while also holding potential applicability in real-time SLAM and robotics domains. We release our software and hardware and self-collected datasets on Github to benefit the community.
1 July 2022 in IEEE Internet of Things Journal
In autonomous driving, environment perception is the fundamental task for intelligent vehicles which provides the necessary environment information for other applications. The main issues in existing environment perception can be categorized into two aspects. On the one hand, all sensors are prone to measurement errors and failures. On the other hand, in complex driving environments, vehicles may encounter a variety of blind spots caused by vehicle occlusions, overlaps, and harsh weather conditions, which will cause sensors to experience low-quality data or to miss crucial environmental information. To cope with these issues, a multivehicle and multisensor (MVMS) cooperative perception method is presented to construct the occupancy grid map (OGM) of vehicles in a global view for the environment perception of autonomous driving. Distinct from existing environment perception methods, our proposed MVMS-OGM not only provides continuous geographical information but also captures and fuses continuous information with soft occupancy probabilities, resulting in more comprehensive and raw environmental information. Simulations and real-world experiments demonstrate that the proposed approach not only expands the perception range in comparison with single-vehicle sensing but also better captures the uncertainty of sensor data by fusing the occupancy probabilities with soft information.
21 October 2024 in IEEE Transactions on Intelligent Vehicles
Accurate modeling of car-following behaviors is crucial for autonomous driving systems. While recent advancements in large language models (LLMs) have shown promise in various domains, their application to car-following tasks has not been fully explored. In this study, we introduce GenFollower, a novel framework that leverages LLMs for car-following modeling. GenFollower employs a specialized prompt engineering technique that integrates heterogeneous inputs into structured natural language prompts, incorporating Chain of Thought (CoT) to enhance reasoning and clarity. GenFollower prioritizes interpretability by explicitly requiring explanations into the prompts, providing valuable insights into the model's decision-making process. Through experiments on the Waymo Open datasets, we demonstrate GenFollower's superior performance and interpretability. Our findings contribute to advancing the field of car-following modeling and pave the way for more robust and reliable autonomous driving systems.
21 October 2022 in IEEE Transactions on Wireless Communications
In this paper, we study a cooperative perception scheme in a vehicular sensor network, attempting to fuse the semantic information provided by different sensors at multiple vehicles, so as to expand the vehicle’s perception range, eliminate blind spots, improve the ability to handle environmental interference and enhance the accuracy and robustness of the perception results. The key to guide the fusion process is the evaluation of the confidence levels of the outputs provided by various machine learning schemes implemented at individual sensors in the vehicular sensor network. We first propose an evaluation criterion termed as Environmental Sensitivity (ES), which is used to measure the sensitivity of the network to environmental changes. Based on the ES, we further evaluate the confidence of the perception output of neural networks and quantify the confidence level considering the abnormal level of the input data, the general performance of the perception algorithm, the detection performance and the ES of the network. Semantic information fusion algorithm is then developed based upon the confidence levels. Experiment results are provided to validate the proposed fusion method in various scenarios.
17 May 2022 in IET Communications
Vehicle behaviour prediction provides important information for decision‐making in modern intelligent transportation systems. People with different driving styles have considerably different driving behaviours and hence exhibit different behaviour tendency. However, most existing prediction methods do not consider the different tendencies in driving styles and apply the same model to all vehicles. Furthermore, most of the existing driver classification methods rely on offline learning that requires a long observation of driving history and hence are not suitable for real‐time driving behaviour analysis. To facilitate personalised models that can potentially improve vehicle behaviour prediction, the authors propose an algorithm that classifies drivers into different driving styles. The algorithm only requires data from a short observation window and it is more applicable for real‐time online applications compared with existing methods that require a long term observation. Experiment results demonstrate that the proposed algorithm can achieve consistent classification results and provide intuitive interpretation and statistical characteristics of different driving styles, which can be further used for vehicle behaviour prediction.