The millimeter wave (mmWave) radio band is promising for the next-generation heterogeneous cellular networks (HetNets) due to its large bandwidth available for meeting the increasing demand of mobile traffic. However, the unique propagation characteristics at mmWave band cause huge redundant handoffs in mmWave HetNets that brings heavy signaling overhead, low energy efficiency and increased user equipment (UE) outage probability if conventional Reference Signal Received Power (RSRP) based handoff mechanism is used. In this paper, we propose a reinforcement learning based handoff policy named SMART to reduce the number of handoffs while maintaining user Quality of Service (QoS) requirements in mmWave HetNets. In SMART, we determine handoff trigger conditions by taking into account both mmWave channel characteristics and QoS requirements of UEs. Furthermore, we propose reinforcement-learning based BS selection algorithms for different UE densities. Numerical results show that in typical scenarios, SMART can significantly reduce the number of handoffs when compared with traditional handoff policies without learning.

Network slicing (NS) has been widely identified as a key architectural technology for 5G-and-beyond systems by supporting divergent requirements in a sustainable way. In radio access network (RAN) slicing, due to the device-base station (BS)-NS three layer association relationship, device association (including access control and handoff management) becomes an essential yet challenging issue. With the increasing concerns on stringent data security and device privacy, exploiting local resources to solve device association problem while enforcing data security and device privacy becomes attractive. Fortunately, recently emerging federated learning (FL), a distributed learning paradigm with data protection, provides an effective tool to address this type of issues in mobile networks. In this paper, we propose an efficient device association scheme for RAN slicing by exploiting a hybrid FL reinforcement learning (HDRL) framework, with the aim to improve network throughput while reducing handoff cost. In our proposed framework, individual smart devices train a local machine learning model based on local data and then send the model features to the serving BS/encrypted party for aggregation, so as to efficiently reduce bandwidth consumption for learning while enforcing data privacy. Specifically, we use deep reinforcement learning to train the local model on smart devices under a hybrid FL framework, where horizontal FL is employed for parameter aggregation on BS, while vertical FL is employed for NS/BS pair selection aggregation on the encrypted party. Numerical results show that the proposed HDRL scheme can achieve significant performance gain in terms of network throughput and communication efficiency incomparison with some state-of-the-art solutions.

The ongoing increasing traffic in the era of big data yields unprecedented demands in user experience and network capacity expansion. The users of next generation mobile networks (5G) should be able to use 3GPP, IEEE, and other technologies simultaneously. The integration of multiple Radio Access Technologies (RATs) of licensed or unlicensed bands has been widely deemed as a cost-efficient way to greatly increase network capacity. In this paper, we propose a Smart Aggregated RAT Access (SARA) strategy with aim of maximizing the long-term network throughput while meeting diverse traffic Quality of Service (QoS) requirements. We consider the scenario that users with different QoS requirements access to a Heterogeneous Network (HetNet) with coexisting Cellular-WiFi. In order to maximize system throughput while meeting diverse traffic QoS requirements in such a complex and dynamic environment, we exploit multi-agent reinforcement learning to perform RAT selection in conjunction with resource allocation for individual user access requests, through sensing dynamic channel states and traffic QoS requirements. In SARA, we first use Nash Q-learning to provide a set of feasible RAT selection strategies while decreasing the strategy space in learning process, and then employ Monte-Carlo Tree Search (MCTS) based Q-learning to perform resource allocation. Numerical results reveal that the network throughput can be maximized while meeting various traffic QoS requirements with limited number of searches by using our proposed SARA algorithm. For bulk arrival access requests, a suboptimal solution can be obtained as high computational complexity is incurred for achieving global optimality. Another attractive feature of SARA is that a tradeoff between the solution optimality and learning time can be readily made by terminating the search of MCTS according to the time constraint. Compared with traditional WiFi offloading schemes, SARA can significantly improve network throughput while guaranteeing traffic QoS requirements.

Mobile crowdsensing (MCS) is a promising paradigm where sensor-embedded mobile devices are exploited for collecting and sharing environmental data. In MCS, the participating mobile devices sense the environment, collect the data, (pre-)process the data and transmit the data or pre-processing results to the server for further processing. In wireless edge networks, transmission and/or processing of sensed data may be unsuccessful due to the unstable wireless channels, limited bandwidth, energy and computation resources. To optimize the MCS performance, it is imperative to jointly consider the data sensing, processing and transmission for MCS system design. In this paper, we propose a joint sensing, communication and computation (JSCC) framework for multi-dimensional resource constrained MCS systems. We formulate the JSCC design as an optimization problem, by jointly controlling the data sensing, transmission and computation offloading schemes in the system. Simulation results show that the proposed JSCC framework significantly outperforms several baseline solutions without jointly considering data sensing-transmission-computation and/or multi-dimensional resource limitations.

Unmanned aerial vehicles (UAVs) are capable of serving as aerial base stations (BSs) for providing dynamic coverage and connectivity extension for the sixth-generation (6G) wireless networks. While flexibility is provided, the deployment of the UAV swarms and the associated resource allocation become rather challenging due to the dynamic nature of UAVs and difficulty in obtaining global user information. In this paper, we propose an adaptive and flexible joint UAV deployment and resource allocation (JUDRA) scheme by exploiting personalized federated deep reinforcement learning, called PFRL, with aim to maximize the long-term network throughput while enforcing user privacy and adapting to time-varying network states. To allow UAVs to make real-time decisions on resource allocation and position adjustment based on local observations while achieving a global optimal solution, a deep reinforcement learning (DRL) algorithm is adopted in the federated learning framework in PFRL. Specifically, we use DRL to train a local model and a personalized model on UAVs, and employ a two-level parameter aggregation scheme on a leading UAV to form a global model. The personalized model can adapt to changing environments, while exploiting the generalization of global model to accelerate the learning convergence. Numerical results show that the proposed PFRL scheme can achieve significant performance gain in terms of network throughput and convergence in comparison with some state-of-art solutions.