In this paper, a joint task, spectrum, and transmit power allocation problem is investigated for a wireless network in which the base stations (BSs) are equipped with mobile edge computing (MEC) servers to jointly provide computational and communication services to users. Each user can request one computational task from three types of computational tasks. Since the data size of each computational task is different, as the requested computational task varies, the BSs must adjust their resource (subcarrier and transmit power) and task allocation schemes to effectively serve the users. This problem is formulated as an optimization problem whose goal is to minimize the maximal computational and transmission delay among all users. A multi-stack reinforcement learning (RL) algorithm is developed to solve this problem. Using the proposed algorithm, each BS can record the historical resource allocation schemes and users’ information in its multiple stacks to avoid learning the same resource allocation scheme and users’ states, thus improving the convergence speed and learning efficiency. Simulation results illustrate that the proposed algorithm can reduce the number of iterations needed for convergence and the maximal delay among all users by up to 18% and 11.1% compared to the standard Q-learning algorithm.

In this article, the problem of vertical handover in software-defined network (SDN) based heterogeneous networks (HetNets) is studied. In the studied model, HetNets are required to offer diverse services for mobile users. Using an SDN controller, HetNets have the capability of managing users' access and mobility issues but still have the problems of ping-pong effect and service interruption during vertical handover. To solve these problems, a mobility-aware seamless handover method based on multipath transmission control protocol (MPTCP) is proposed. The proposed handover method is executed in the controller of the software-defined HetNets (SDHetNets) and consists of three steps: location prediction, network selection, and handover execution. In particular, the method first predicts the user's location in the next moment with an echo state network (ESN). Given the predicted location, the SDHetNet controller can determine the candidate network set for the handover to pre-allocate network wireless resources. Second, the target network is selected through fuzzy analytic hierarchical process (FAHP) algorithm, jointly considering user preferences, service requirements, network attributes, and user mobility patterns. Then, seamless handover is realized through the proposed MPTCP-based handover mechanism. Simulations using real-world user trajectory data from Korea Advanced Institute of Science & Technology show that the proposed method can reduce the handover times by 10.85% to 29.12% compared with traditional methods. The proposed method also maintains at least one MPTCP subflow connected during the handover process and achieves a seamless handover.

In this paper, the problem of trajectory design for unmanned aerial vehicle (UAV) swarms in data collection Internet of Things (IoT) networks is studied. In the considered model, the UAV swarm is deployed to patrol a designated area and collect status information from sensors monitoring physical processes. The sense-collect-interchange-explore (SCIE) protocol is proposed to regulate UAV actions, ensuring synchronization and adaptability in a distributed manner. To maintain real-time monitoring while reducing data transmission, we introduce status freshness, which is an extension of age of information (AoI) and allows negative values to reflect the swarm’s predictive capabilities. The trajectory design problem is then formulated as an optimization problem to minimize average status freshness. A virtual force-based algorithm is developed to solve this problem, where UAVs are influenced by attractive forces from sensors and repulsive forces from neighbors. These forces guide UAVs toward sensors requiring data transmission while reducing communication overlap. The proposed distributed algorithm allows each UAV to independently design its trajectory, reducing redundancy and enhancing scalability. Simulation results show that the proposed method can significantly reduce average status freshness under the same energy efficiency conditions compared to artificial potential field algorithm. The proposed method also achieves significantly reduction in terms of communication overhead, compared to fully connected strategies, ensuring scalability in large-scale UAV deployments.

In this paper, the problem of trajectory design for energy harvesting unmanned aerial vehicles (UAVs) is studied. In the considered model, the UAV acts as a moving base station to serve the ground users, while collecting energy from the charging stations located at the center of a user group. For this purpose, the UAV must be examined and repaired regularly. In consequence, it is necessary to optimize the trajectory design of the UAV while jointly considering the maintenance costs, the reward of serving users, the energy management, and the user service time. To capture the relationship among these factors, we first model the completion of service and the harvested energy as the reward, and the energy consumption during the deployment as the cost. Then, the deployment profitability is defined as the ratio of the reward to the cost of the UAV trajectory. Based on this definition, the trajectory design problem is formulated as an optimization problem whose goal is to maximize the deployment profitability of the UAV. To solve this problem, a foraging-based algorithm is proposed to find the optimal trajectory so as to maximize the deployment profitability and minimize the average user service time. The proposed algorithm can find the optimal trajectory for the UAV with low time complexity at the level of polynomial. Fundamental analysis shows that the proposed algorithm achieves the maximal deployment profitability. Simulation results show that, compared to Q-learning algorithm, the proposed algorithm effectively reduces the operation time and the average user service time while achieving the maximal deployment profitability.

In this paper, the problem of trajectory design of unmanned aerial vehicles (UAVs) for maximizing the number of satisfied users is studied in a UAV based cellular network where the UAV works as a flying base station that serves users, and the user indicates its satisfaction in terms of completion of its data request within an allowable maximum waiting time. The trajectory design is formulated as an optimization problem whose goal is to maximize the number of satisfied users. To solve this problem, a machine learning framework based on double Q-learning algorithm is proposed. The algorithm enables the UAV to find the optimal trajectory that maximizes the number of satisfied users. Compared to the traditional learning algorithms, such as Q-learning that selects and evaluates the action using the same Q-table, the proposed algorithm can decouple the selection from the evaluation, therefore avoid overestimation which leads to sub-optimal policies. Simulation results show that the proposed algorithm can achieve up to 19.4% and 14.1% gains in terms of the number of satisfied users compared to random algorithm and Q-learning algorithm.