The recent concept of massive multiple-input multiple-output (MIMO) can significantly improve the capacity of the communication network, and it has been regarded as a promising technology for the next-generation wireless communications. However, the fundamental challenge of existing massive MIMO systems is that high computational complexity and complicated spatial structures bring great difficulties to exploit the characteristics of the channel and sparsity of these multi-antennas systems. To address this problem, in this paper, we focus on channel estimation and direction-of-arrival (DOA) estimation, and a novel framework that integrates the massive MIMO into deep learning is proposed. To realize end-to-end performance, a deep neural network (DNN) is employed to conduct offline learning and online learning procedures, which is effective to learn the statistics of the wireless channel and the spatial structures in the angle domain. Concretely, the DNN is first trained by simulated data in different channel conditions with the aids of the offline learning, and then corresponding output data can be obtained based on current input data during online learning process. In order to realize super-resolution channel estimation and DOA estimation, two algorithms based on the deep learning are developed, in which the DOA can be estimated in the angle domain without additional complexity directly. Furthermore, simulation results corroborate that the proposed deep learning based scheme can achieve better performance in terms of the DOA estimation and the channel estimation compared with conventional methods, and the proposed scheme is well investigated by extensive simulation in various cases for testing its robustness.

Automatic modulation recognition (AMR) is an essential and challenging topic in the development of the cognitive radio (CR), and it is a cornerstone of CR adaptive modulation and demodulation capabilities to sense and learn environments and make corresponding adjustments. AMR is essentially a classification problem, and deep learning achieves outstanding performances in various classification tasks. So, this paper proposes a deep learning-based method, combined with two convolutional neural networks (CNNs) trained on different datasets, to achieve higher-accuracy AMR. A CNN is trained on samples composed of in-phase and quadrature component signals, otherwise known as in-phase and quadrature (IQ) samples, to distinguish modulation modes that are relatively easy to identify. We adopt dropout instead of pooling operation to achieve higher recognition accuracy. A CNN based on constellation diagrams is also designed to recognize modulation modes that are difficult to distinguish in the former CNN, such as 16QAM and 64QAM, demonstrating the ability to classify QAM signals even in scenarios with a low signal-to-noise ratio.

In the fifth-generation (5G) mobile communication system, various service requirements of different communication environments are expected to be satisfied. As a new evolution network structure, heterogeneous network (HetNet) has been studied in recent years. Compared with homogeneous networks, HetNets can increase the opportunity in the spatial resource reuse and improve users' quality of service by developing small cells into the coverage of macrocells. Since there is mutual interference among different users and the limited spectrum resource in HetNets, however, efficient resource allocation (RA) algorithms are vitally important to reduce the mutual interference and achieve spectrum sharing. In this article, we provide a comprehensive survey on RA in HetNets for 5G communications. Specifically, we first introduce the definition and different network scenarios of HetNets. Second, RA models are discussed. Then, we present a classification to analyze current RA algorithms for the existing works. Finally, some challenging issues and future research trends are discussed. Accordingly, we provide two potential structures for 6G communications to solve the RA problems of the next-generation HetNets, such as a learning-based RA structure and a control-based RA structure. The goal of this article is to provide important information on HetNets, which could be used to guide the development of more efficient techniques in this research area.

Non-orthogonal multiple access (NOMA) has been considered as an essential multiple access technique for enhancing system capacity and spectral efficiency in future communication scenarios. However, the existing NOMA systems have a fundamental limit: high computational complexity and a sharply changing wireless channel make exploiting the characteristics of the channel and deriving the ideal allocation methods very difficult tasks. To break this fundamental limit, in this paper, we propose a novel and effective deep learning (DL)-aided NOMA system, in which several NOMA users with random deployment are served by one base station (BS). Since DL is advantageous in that it allows training the input signals and detecting sharply changing channel conditions, we exploit it to address wireless NOMA channels in an end-to-end manner. Specifically, it is employed in the proposed NOMA system to learn a completely unknown channel environment. A long shortterm memory (LSTM) network based on DL is incorporated into a typical NOMA system, enabling the proposed scheme to detect the channel characteristics automatically. In the proposed strategy, the LSTM is first trained by simulated data under different channel conditions via offline learning, and then the corresponding output data can be obtained based on the current input data used during the online learning process. In general, we build, train and test the proposed cooperative framework to realize automatic encoding, decoding and channel detection in an additive white Gaussian noise (AWGN) channel. Furthermore, we regard one conventional user activity and data detection scheme as an unknown nonlinear mapping operation and use LSTM to approximate it to evaluate the data detection capacity of DL based on NOMA. Simulation results demonstrate that the proposed scheme is robust and efficient compared with conventional approaches. In addition, the accuracy of the LSTMaided NOMA scheme is studied by introducing the well-known 10-fold cross-validation procedure.

Millimeter wave (mmWave) massive multiple-input multiple-output (MIMO) has been regarded to be an emerging solution for the next generation of communications, in which hybrid analog and digital precoding is an important method for reducing the hardware complexity and energy consumption associated with mixed signal components. However, the fundamental limitations of the existing hybrid precoding schemes is that they have high computational complexity and fail to fully exploit the spatial information. To overcome these limitations, this paper proposes a deep-learning-enabled mmWave massive MIMO framework for effective hybrid precoding, in which each selection of the precoders for obtaining the optimized decoder is regarded as a mapping relation in the deep neural network (DNN). Specifically, the hybrid precoder is selected through training based on the DNN for optimizing precoding process of the mmWave massive MIMO. Additionally, we present extensive simulation results to validate the excellent performance of the proposed scheme. The results exhibit that the DNN-based approach is capable of minimizing the bit error ratio (BER) and enhancing spectrum efficiency of the mmWave massive MIMO, which achieves better performance in hybrid precoding compared with conventional schemes while substantially reducing the required computational complexity.

Deep learning (DL) has been widely proposed for botnet attack detection in Internet of Things (IoT) networks. However, the traditional centralized DL (CDL) method cannot be used to detect the previously unknown (zero-day) botnet attack without breaching the data privacy rights of the users. In this article, we propose the federated DL (FDL) method for zero-day botnet attack detection to avoid data privacy leakage in IoT-edge devices. In this method, an optimal deep neural network (DNN) architecture is employed for network traffic classification. A model parameter server remotely coordinates the independent training of the DNN models in multiple IoT-edge devices, while the federated averaging (FedAvg) algorithm is used to aggregate local model updates. A global DNN model is produced after a number of communication rounds between the model parameter server and the IoT-edge devices. The zero-day botnet attack scenarios in IoT-edge devices is simulated with the Bot-IoT and N-BaIoT data sets. Experiment results show that the FDL model: 1) detects zero-day botnet attacks with high classification performance; 2) guarantees data privacy and security; 3) has low communication overhead; 4) requires low-memory space for the storage of training data; and 5) has low network latency. Therefore, the FDL method outperformed CDL, localized DL, and distributed DL methods in this application scenario.