Federated learning (FL) is a viable technique to train a shared machine learning model without sharing data. Hierarchical FL (HFL) system has yet to be studied regrading its multiple levels of energy, computation, communication, and client scheduling, especially when it comes to clients relying on energy harvesting to power their operations. This paper presents a new two-phase deep deterministic policy gradient (DDPG) framework, referred to as “TP-DDPG”, to balance online the learning delay and model accuracy of an FL process in an energy harvesting-powered HFL system. The key idea is that we divide optimization decisions into two groups, and employ DDPG to learn one group in the first phase, while interpreting the other group as part of the environment to provide rewards for training the DDPG in the second phase. Specifically, the DDPG learns the selection of participating clients, and their CPU configurations and the transmission powers. A new straggler-aware client association and bandwidth allocation (SCABA) algorithm efficiently optimizes the other decisions and evaluates the reward for the DDPG. Experiments demonstrate that with substantially reduced number of learnable parameters, the TP-DDPG can quickly converge to effective polices that can shorten the training time of HFL by 39.4% compared to its benchmarks, when the required test accuracy of HFL is 0.9.

Mobile edge computing (MEC) offers an opportunity for devices relying on wireless power transfer (WPT), to accomplish computationally demanding tasks. Such WPT-powered MEC systems have yet to be optimized for long-term efficiency, due to random and changing task demands and wireless channel states of the devices. This paper presents an augmented two-staged deep Q-network (DQN), referred to as “TS-DQN”, for online optimization of WPT-powered MEC systems, where the WPT, offloading schedule, channel allocation, and the CPU configurations of the edge server and devices are jointly optimized to minimize the long-term average energy requirement of the systems. The key idea is to design a DQN for learning the channel allocation and task admission, while the WPT, offloading time and CPU configurations are efficiently optimized to precisely evaluate the reward of the DQN and substantially reduce its action space. Another important aspect is that a new action generation method is developed to expand and diversify the actions of the DQN, further accelerating its convergence. As validated by simulations, the proposed TS-DQN is much more energy efficient and converges much faster, than its potential alternative directly using the state-of-the-art Deep Deterministic Policy Gradient algorithm to learn all decision variables.

Extended reality (XR) is one of the most important applications of beyond 5G and 6G networks. Real-time XR video transmission presents challenges in terms of data rate and delay. In particular, the frame-by-frame transmission mode of XR video makes real-time XR video very sensitive to dynamic network environments. To improve the users’ quality of experience (QoE), we design a cross-layer transmission framework for real-time XR video. The proposed framework allows the simple information exchange between the base station (BS) and the XR server, which assists in adaptive bitrate and wireless resource scheduling. We utilize the cross-layer information to formulate the problem of maximizing user QoE by finding the optimal scheduling and bitrate adjustment strategies. To address the issue of mismatched time scales between two strategies, we decouple the original problem and solve them individually using a multi-agent-based approach. Specifically, we propose the multi-step Deep Q-network (MS-DQN) algorithm to obtain a frame-priority-based wireless resource scheduling strategy and then propose the Transformer-based Proximal Policy Optimization (TPPO) algorithm for video bitrate adaptation. The experimental results show that the TPPO+MS-DQN algorithm proposed in this study can improve the QoE by 3.6% to 37.8%. More specifically, the proposed MS-DQN algorithm enhances the transmission quality by 49.9%-80.2%.

Extended Reality (XR) is an important service in the 5G network and in future 6G networks. In contrast to traditional video on demand services, real-time XR video is transmitted frame-by-frame, requiring low latency and being highly sensitive to network fluctuations. In this paper, we model the quality of experience (QoE) for real-time XR video transmission on a frame-by-frame basis. Based on the proposed QoE model, we formulate an optimization problem that maximizes QoE with constraints on wireless resources and long-term energy consumption. We utilize Lyapunov optimization to transform the original problem into a single-frame optimization problem and then allocate wireless subchannels. We propose an adaptive XR video bitrate algorithm that employs a Long Short Term Memory (LSTM) based Deep Q-Network (DQN) algorithm for video bitrate selection. Through numerical results, we show that our proposed algorithm outperforms the baseline algorithms, with the average QoE improvements of 0.04 to 0.46. Specifically, compared to baseline algorithms, the proposed algorithm reduces average video quality variations by 29% to 50% and improves the frame transmission success rate by 5% to 48%.

Powering mobile edge computing (MEC) with a hybrid supply of smart grid (SG) and renewable energy source (RES) offers an opportunity to utilize clean energy and cut down energy expenses under two-way energy transactions. We propose two-timescale online resource allocation and energy management (TSRE) for MEC with a hybrid power supply, adapting to dynamic task and RES arrivals, wireless channels, and energy prices. The TSRE minimizes the time-averaged cost of predictive energy planning and real-time energy trading of base stations (BSs), and the energy usage of mobile users. By generalizing the Lyapunov optimization and stochastic subgradient method, the energy planning (sub)problem is solved upon RES arrivals using only historical data. The real-time offloading schedules, energy trading decisions and CPU configurations are decoupled over time and (asymptotically) optimally made in a distributed manner. The feasibility and the asymptotic optimality of the TSRE are proved. Numerical results demonstrate that the TSRE saves system cost significantly by 33.7%, compared to its baselines.

Heterogeneous networks, constituted by conventional macro cells and overlaying pico cells, have been deemed a promising paradigm to support the deluge of data traffic with higher spectral efficiency and Energy Efficiency (EE). In order to deploy pico cells in reality, the density of Pico Base Stations (PBSs) and the pico Cell Range Expansion (CRE) are two important factors for the network spectral efficiency as well as EE improvement. However, associated with the range and density evolution, the inter-tier interference within the heterogeneous architecture will be challenging, and the time domain Enhanced Inter-cell Interference Coordination (eICIC) technique becomes necessary. Aiming to improve the network EE, the above factors are jointly considered in this paper. More specifically, we first derive the closed-form expression of the network EE as a function of the density of PBSs and pico CRE bias based on stochastic geometry theory, followed by a linear search algorithm to optimize the pico CRE bias and PBS density, respectively. Moreover, in order to realize the pico CRE bias and PBS density joint optimization, a heuristic algorithm is proposed to achieve the network EE maximization. Numerical simulations show that our proposed pico CRE bias and PBS density joint optimization algorithm can improve the network EE significantly with low computational complexity.