With autonomous driving developing in a booming stage, accurate object detection in complex scenarios attract wide attention to ensure the safety of autonomous driving. Millimeter wave (mmWave) radar and vision fusion is a mainstream solution for accurate obstacle detection. This article presents a detailed survey on mmWave radar and vision fusion based obstacle detection methods. First, we introduce the tasks, evaluation criteria, and datasets of object detection for autonomous driving. The process of mmWave radar and vision fusion is then divided into three parts: sensor deployment, sensor calibration, and sensor fusion, which are reviewed comprehensively. Specifically, we classify the fusion methods into data level, decision level, and feature level fusion methods. In addition, we introduce three-dimensional(3D) object detection, the fusion of lidar and vision in autonomous driving and multimodal information fusion, which are promising for the future. Finally, we summarize this article.

For autonomous driving, it is important to detect obstacles in all scales accurately for safety consideration. In this paper, we propose a new spatial attention fusion (SAF) method for obstacle detection using mmWave radar and vision sensor, where the sparsity of radar points are considered in the proposed SAF. The proposed fusion method can be embedded in the feature-extraction stage, which leverages the features of mmWave radar and vision sensor effectively. Based on the SAF, an attention weight matrix is generated to fuse the vision features, which is different from the concatenation fusion and element-wise add fusion. Moreover, the proposed SAF can be trained by an end-to-end manner incorporated with the recent deep learning object detection framework. In addition, we build a generation model, which converts radar points to radar images for neural network training. Numerical results suggest that the newly developed fusion method achieves superior performance in public benchmarking. In addition, the source code will be released in the GitHub.

Automatic modulation classification (AMC) is to identify the modulation type of a received signal, which plays a vital role to ensure the physical-layer security for Internet of Things (IoT) networks. Inspired by the great success of deep learning in pattern recognition, the convolutional neural network (CNN) and recurrent neural network (RNN) are introduced into the AMC. In general, there are two popular data formats used by AMC, which are the in-phase/quadrature (I/Q) representation and amplitude/phase (A/P) representation, respectively. However, most of AMC algorithms aim at structure innovations, while the differences and characteristics of I/Q and A/P are ignored to analyze. In this article, lots of popular AMC algorithms are reproduced and evaluated on the same data set, where the I/Q and A/P are used, respectively, for comparison. Based on the experimental results, it is found that: 1) CNN-RNN-like algorithms using A/P as input data are superior to those using I/Q at high signal-to-noise ratio (SNR), while it has an opposite result in low SNR and 2) the features extracted from I/Q and A/P are complementary to each other. Motivated by the aforementioned findings, a multitask learning-based deep neural network (MLDNN) is proposed, which effectively fuses I/Q and A/P. In addition, the MLDNN also has a novel backbone, which is made up of three blocks to extract discriminative features, and they are CNN block, bidirectional gated recurrent unit (BiGRU) block, and a step attention fusion network (SAFN) block. Different from most of CNN-RNN-like algorithms (i.e., they only use the last step outputs of RNN), all step outputs of BiGRU can be effectively utilized by MLDNN with the help of SAFN. Extensive simulations are conducted to verify that the proposed MLDNN achieves superior performance in the public benchmark.

Radio frequency fingerprint identification (RFFI) is a promising physical layer security technique that employs the hardware-introduced features extracted from the received signals for device identification. In this paper, we consider an RFFI problem in the presence of hybrid time-varying distortions (HTVDs) induced by multipath fading channel, carrier frequency offset (CFO), and phase offset. To solve this problem, an HTVDs-robust RFFI framework is proposed. Firstly, we derive that the residual HTVDs after CFO correction can be approximated as multiplicative interference in the frequency domain. Secondly, we define a novel signal analysis dimension named spectral quotient (SQ) representation and then present the spectral circular shift division (SCSD) method to generate the HTVDs-robust SQ signals, where the multiplicative interference can be suppressed. Thereafter, the statistics including root mean square (RMS), variance (VAR), skewness (SKE), and kurtosis (KUR) are extracted from the real and imaginary components of the SQ signals, respectively. Finally, the statistical features are used for the training and testing of the support vector machine (SVM) classifiers. To further enhance the performance of the proposed RFFI scheme, we also present the spectral circular multi-shift division (SCMSD) method, which increases the flexibility in the generation of the HTVDs-robust SQ signals. Given what we knew, this is the first time attempting to mitigate the HTVDs by leveraging the strong frequency correlation at the neighboring subcarriers in the multivariate hypothesis tasks. Compared to several handcraft feature-based RFFI methods, the proposed method exhibits superior identification accuracy and strong robustness. Experimental results show that the proposed RFFI scheme can achieve the accuracy of 91.3%with five devices and 86.4% with sixteen devices when the classifiers are trained with the additive white Gaussian noise but are tested with the Rayleigh channel.

For future connected automated vehicles (CAVs) networks, the joint optimization of communication, sensing, and computing resources is crucial to guarantee the performance of cooperative automated driving’s safety, which is attracting more and more attention. However, the existing works have not considered the low-latency requirement for the raw perception data sharing with both wireless communication link capability and computing efficiency constraints, causing a serious threat to the cooperative automated driving’s safety in CAVs networks. In this article, a vehicle–road–base station cooperation architecture is designed, and a federated reinforcement learning (FRL)-based task offloading and resource allocation algorithm in the CAVs network is proposed to reduce the task execution delay with different communication and computing constraints. The problem of execution delay minimization is theoretically formulated and analyzed under three task practical offloading modes. To adapt to the dynamic topology of the CAVs network, we design a deep reinforcement learning algorithm to achieve the optimal task offloading and resource allocation. To further reduce the data transmission overhead of the centralized reinforcement learning algorithm, the FRL-enabled algorithm is proposed to minimize the execution delay of the optimal task offloading and resource allocation among multiple CAVs. Both the simulation and hardware testbed results verify that the proposed algorithms can not only reduce the execution delay and the communication overhead but also improve the system throughput.

Automatic modulation classification (AMC) identifies a received signal’s modulation scheme without prior knowledge of the intercepted signal, which enables significant applications in both the military and civilian domains. Inspired by the great success of deep learning (DL), lots of neural networks are introduced into AMC. To further improve classification performance, various complementary cues including in-phase/quadrature (I/Q), amplitude/phase (A/P), constellation, and other formats are used together to enhance the discrimination of the DL model, where only outputs of the last layer are used. In this paper, we find that different layers’ outputs in the DL model are also complementary to each other. As a result, a hierarchical classification head based convolutional gated deep neural network (HCGDNN) is proposed by utilizing different layers’ output, which only uses the I/Q cue. The proposed HCGDNN consists of three groups of convolutional neural networks (CNN) blocks, two groups of bidirectional gated recurrent units (BiGRU), and a hierarchical classification head. Compared to the long short-term memory (LSTM), the BiGRU has a smaller computational complexity and also releases the gradient dispersion and explosion in the training phase. With the help of the hierarchical classification head, three groups of modulation predictions are made for a received I/Q signal. After that, a novel nonlinear optimization fusion method is derived to generate fusion weights to fuse different groups, then a final classification decision is made. Compared to AMC methods using various cues, the proposed HCGDNN only uses I/Q cue and has low computational overhead. Numerical results suggest that the newly developed HCGDNN achieves superior performance on the public benchmark.To help other researchers, the source code will be uploaded to the github as long as the paper is published.