The fifth generation (5G) network will serve as a key enabler in meeting the continuously increasing demands for future wireless applications, including an ultra-high data rate, an ultrawide radio coverage, an ultra-large number of devices, and an ultra-low latency. This article examines security, a pivotal issue in the 5G network where wireless transmissions are inherently vulnerable to security breaches. Specifically, we focus on physical layer security, which safeguards data confidentiality by exploiting the intrinsic randomness of the communications medium and reaping the benefits offered by the disruptive technologies to 5G. Among various technologies, the three most promising ones are discussed: heterogenous networks, massive multiple-input multiple-output, and millimeter wave. On the basis of the key principles of each technology, we identify the rich opportunities and the outstanding challenges that security designers must tackle. Such an identification is expected to decisively advance the understanding of future physical layer security.

We propose and analyze transmit antenna selection (TAS) to enhance physical layer security in a wiretap channel with N _A antennas at the transmitter, N _B antennas at the receiver, and N _E antennas at the eavesdropper. We focus on the practical scenario where the transmitter does not have any channel state information (CSI) of the eavesdropper's channel. The transmitter selects a single antenna that maximizes the instantaneous signal-to-noise ratio (SNR) at the receiver. The receiver and the eavesdropper employ either maximal-ratio combining (MRC) or selection combining (SC) to combine the received signals. For the proposed protocols, we derive new closed-form expressions for the probability of non-zero secrecy capacity. We consider Nakagami-m fading with non-identical fading parameters of the main channel, m _B , and of the eavesdropper's channel, m _E . Next, we derive new closed-form expressions for the exact secrecy outage probability, based on which the ε-outage secrecy capacity is characterized. Based on the exact expressions, we derive the asymptotic secrecy outage probability which accurately reveals the secrecy diversity order and the secrecy array gain. We confirm that the proposed protocols achieve identical secrecy diversity orders of N _A N _B m _B . An interesting conclusion is reached that this diversity order is independent of N _E and m _E . Furthermore, we prove that under the proposed protocols, the secrecy outage probability and the ε-outage secrecy capacity improve with increasing N _A .

This work introduces downlink non-orthogonal multiple access (NOMA) into short-packet communications. NOMA has great potential to improve fairness and spectral efficiency with respect to orthogonal multiple access (OMA) for low-latency downlink transmission, thus making it attractive for the emerging Internet of Things. We consider a two-user downlink NOMA system with finite blocklength constraints, in which the transmission rates and power allocation are optimized. To this end, we investigate the trade-off among the transmission rate, decoding error probability, and the transmission latency measured in blocklength. Then, a one-dimensional search algorithm is proposed to resolve the challenges mainly due to the achievable rate affected by the finite blocklength and the unguaranteed successive interference cancellation. We also analyze the performance of OMA as a benchmark to fully demonstrate the benefit of NOMA. Our simulation results show that NOMA significantly outperforms OMA in terms of achieving a higher effective throughput subject to the same finite blocklength constraint, or incurring a lower latency to achieve the same effective throughput target. Interestingly, we further find that with the finite blocklength, the advantage of NOMA relative to OMA is more prominent when the effective throughput targets at the two users become more comparable.

This paper proposes a new design of non-orthogonal multiple access (NOMA) under secrecy considerations. We focus on a NOMA system, where a transmitter sends confidential messages to multiple users in the presence of an external eavesdropper. The optimal designs of decoding order, transmission rates, and power allocated to each user are investigated. Considering the practical passive eavesdropping scenario where the instantaneous channel state of the eavesdropper is unknown, we adopt the secrecy outage probability as the secrecy metric. We first consider the problem of minimizing the transmit power subject to the secrecy outage and quality of service constraints, and derive the closed-form solution to this problem. We then explore the problem of maximizing the minimum confidential information rate among users subject to the secrecy outage and transmit power constraints, and provide an iterative algorithm to solve this problem. We find that the secrecy outage constraint in the studied problems does not change the optimal decoding order for NOMA, and one should increase the power allocated to the user whose channel is relatively bad when the secrecy constraint becomes more stringent. Finally, we show the advantage of NOMA over orthogonal multiple access in the studied problems both analytically and numerically.

We analyze the impact of antenna correlation on secrecy performance of multiple-input multiple-output wiretap channels where transmitter employs transmit antenna selection while receiver and eavesdropper perform maximal-ratio combining with arbitrary correlation. New closed-form expressions are derived for the exact and asymptotic (high signal-to-noise ratio in transmitter-receiver channel) secrecy outage probability.

We investigate the impact of cochannel interference on the security performance of multiple amplify-and-forward (AF) relaying networks, where N intermediate AF relays assist the data transmission from the source to the destination. The relays are corrupted by multiple cochannel interferers, and the information transmitted from the relays to the destination can be overheard by the eavesdropper. In order to deal with the interference and wiretap, the best out of N relays is selected for security enhancement. To this end, we derive a novel lower bound on the secrecy outage probability (SOP), which is then utilized to present two best relay selection criteria, based on the instantaneous and statistical channel information of the interfering links. For these criteria and the conventional max-min criterion, we quantify the impact of cochannel interference and relay selection by deriving the lower bound on the SOP. Furthermore, we derive the asymptotic SOP for each criterion, to explicitly reveal the impact of transmit power allocation among interferers on the secrecy performance, which offers valuable insights into practical design. We demonstrate that the all selection criteria achieve full secrecy diversity order N, while the proposed two criteria in this paper outperform the conventional max-min scheme.