Future Internet involves several emerging technologies such as 5G and beyond 5G networks, vehicular networks, unmanned aerial vehicle (UAV) networks, and Internet of Things (IoTs). Moreover, the future Internet becomes heterogeneous and decentralized with a large number of involved network entities. Each entity may need to make its local decision to improve the network performance under dynamic and uncertain network environments. Standard learning algorithms such as single-agent Reinforcement Learning (RL) or Deep Reinforcement Learning (DRL) have been recently used to enable each network entity as an agent to learn an optimal decision-making policy adaptively through interacting with the unknown environments. However, such an algorithm fails to model the cooperations or competitions among network entities, and simply treats other entities as a part of the environment that may result in the non-stationarity issue. Multi-agent Reinforcement Learning (MARL) allows each network entity to learn its optimal policy by observing not only the environments but also other entities’ policies. As a result, MARL can significantly improve the learning efficiency of the network entities, and it has been recently used to solve various issues in the emerging networks. In this paper, we thus review the applications of MARL in emerging networks. In particular, we provide a tutorial of MARL and a comprehensive survey of applications of MARL in next-generation Internet. In particular, we first introduce single-agent RL and MARL. Then, we review a number of applications of MARL to solve emerging issues in the future Internet. The issues consist of network access, transmit power control, computation offloading, content caching, packet routing, trajectory design for UAV-aided networks, and network security issues. Finally, we discuss the challenges, open issues, and future directions related to the applications of MARL in the future Internet.

As one of the most promising applications in future Internet of Things, Internet of Vehicles (IoV) has been acknowledged as a fundamental technology for developing the Intelligent Transportation Systems in smart cities. With the emergence of the sixth generation (6G) communications technologies, massive network infrastructures will be densely deployed and the number of network nodes will increase exponentially, leading to extremely high energy consumption. There has been an upsurge of interest to develop the green IoV towards sustainable vehicular communication and networking in the 6G era. However, as a special mobile ad-hoc network, the energy cost in an IoV system involves the communication and computation energy in addition to the fuel consumption and the electricity cost of moving vehicles. Moreover, the energy harvesting technology, which is likely to be adopted widely in 6G systems, will complicate the optimization of energy efficiency in the entire system. Current studies focus only on part of the energy issues in IoV systems without a comprehensive discussion of the state-of-the-art energy-efficient approaches and the influence of the development of 6G networks on green IoV. In this paper, we present the main considerations for green IoV from five different scenarios, including the communication, computation, traffic, Electric Vehicles (EVs), and energy harvesting management. The literature relevant to each of the scenarios is compared from the perspective of energy optimization (e.g., with respect to resource allocation, workload scheduling, routing design, traffic control, charging management, energy harvesting and sharing) and the related factors affecting energy efficiency (e.g., resource limitation, channel state, network topology, traffic condition). In addition, we introduce the potential challenges and the emerging technologies in 6G for developing green IoV systems. Finally, we discuss the research trends in designing energy-efficient IoV systems.

Federated learning (FL) becomes popular and has shown great potentials in training large-scale machine learning (ML) models without exposing the owners’ raw data. In FL, the data owners can train ML models based on their local data and only send the model updates rather than raw data to the model owner for aggregation. To improve learning performance in terms of model accuracy and training completion time, it is essential to recruit sufficient participants. Meanwhile, the data owners are rational and may be unwilling to participate in the collaborative learning process due to the resource consumption. To address the issues, there have been various works recently proposed to motivate the data owners to contribute their resources. In this paper, we provide a comprehensive review for the economic and game theoretic approaches proposed in the literature to design various schemes for incentivizing data owners to participate in FL training process. In particular, we first present the fundamentals and background of FL, economic theories commonly used in incentive mechanism design. Then, we review applications of game theory and economic approaches applied for incentive mechanisms design of FL. Finally, we highlight some open issues and future research directions concerning incentive mechanism design of FL.

Metaverse, also known as the Internet of 3-D worlds, has recently attracted much attention from both academia and industry. Each virtual subworld, operated by a virtual service provider (VSP), provides a type of virtual service. Digital twins (DTs), namely, digital replicas of physical objects, are key enablers. Generally, a DT belongs to the party that develops it and establishes the communication link between the two worlds. However, in an interoperable metaverse, data-like DTs can be “shared” within the platform. Therefore, one set of DTs can be leveraged by multiple VSPs. As the quality of the shared DTs may not always be satisfying, in this article, we propose an agile solution, i.e., a dynamic hierarchical framework, in which a group of Internet of Things devices in the lower level are incentivized to collectively sense physical objects’ status information and VSPs in the upper level determine synchronization intensities to maximize their payoffs. We adopt an evolutionary game approach to model the devices VSP selections and a simultaneous differential game to model the optimal synchronization intensity control problem. We further extend it as a Stackelberg differential game by considering some VSPs to be first movers. We provide open-loop solutions based on the control theory for both formulations. We theoretically and experimentally show the existence, uniqueness, and stability of the equilibrium to the lower level game and further provide a sensitivity analysis for various system parameters. Experiments show that the proposed dynamic hierarchical game outperforms the baseline.

Unmanned aerial vehicles (UAVs) have been introduced into wireless communication systems to provide high-quality services and enhanced coverage due to their high mobility. In this paper, we study a UAV-assisted mobile edge computing (MEC) system in which a moving UAV equipped with computing resources is employed to help user devices (UDs) computing their tasks. The computing tasks of each UD can be divided into two parts: one portion is processed locally and the remaining portion is offloaded to the UAV for computing. Offloading is enabled by uplink and downlink communications between UDs and the UAV. On this basis, two types of access modes are considered, namely non-orthogonal and orthogonal multiple access. For both access modes, we formulate new optimization problems to minimize the weighted sum energy consumption of the UAV and UDs by jointly optimizing the UAV trajectory and computation resource allocation, under the constraint on the number of computation bits. These problems are non-convex optimization problems that are difficult to solve directly. Accordingly, we develop alternating iterative algorithms to solve them based on the block alternating descent method. Specifically, the UAV trajectory and computation resource allocation are alteratively optimized in each iteration. Extensive simulation results demonstrate the significant energy savings of our proposed joint design over the benchmarks.

It is well known that unmanned aerial vehicles (UAVs) can help terrestrial base stations (BSs) offload data traffic from crowded areas to improve coverage and boost throughput. However, the limited backhaul capacity cannot cope with the ever-increasing data demands, for which caching is introduced to relieve the backhaul bottleneck. In this paper, we focus on a multi-UAV assisted wireless network, and target to fully utilize the benefits of wireless caching and UAV mobility for multiuser content delivery. By taking into account the limited storage, our goal is to maximize the minimum throughput among UAV-served users by jointly optimizing cache placement, UAV trajectory, and transmission power in a finite period. The resultant problem is a mixed-integer non-convex optimization problem. To facilitate solving this problem, an alternating iterative algorithm is proposed by adopting the block alternating descent and successive convex approximation methods. Specifically, this problem is split into three subproblems, namely cache placement optimization, trajectory optimization, and power allocation optimization. Then these subproblems are solved alternately in an iterative manner. We show that the proposed algorithm can converge to the set of stationary solutions of this problem. Besides, we further analyze the computational complexity of this algorithm. Numerical results show that great throughput enhancement is achieved by applying our proposed joint design in comparison with other benchmarks without trajectory design and power control.