Metaverse is an artificial virtual world mapped from and interacting with the real world. In metaverse, digital entities coexist with their physical counterparts. Powered by deep learning, metaverse is inevitably becoming more intelligent in the interactions between reality and virtuality. However, it is confronted with a nontrivial problem known as sim2real transfer when deep learning techniques try to bridge the reality gap between the physical world and simulations. In this article, we use multiagent deep reinforcement learning (MARL) to implement collective intelligence for digital entities as well as their physical counterparts. To model the immersive environments in metaverse, we define a nonstationary variant of Markov games and propose a recurrent MARL solution to it. Based on the solution, MARL sim2real transfer that bridges real and virtual multiple unmanned aerial vehicle (multi-UAV) systems is successfully conducted by employing recurrent multiagent deep deterministic policy gradient (R-MADDPG) with the domain randomization technique. Additionally, we use perception-control modularization to improve the generalization performance of MARL policies and make training more efficient.

Multiagent deep reinforcement learning (DRL) makes optimal decisions dependent on system states observed by agents, but any uncertainty on the observations may mislead agents to take wrong actions. The mean-field actor-critic (MFAC) reinforcement learning is well-known in the multiagent field since it can effectively handle a scalability problem. However, it is sensitive to state perturbations that can significantly degrade the team rewards. This work proposes a Robust MFAC (RoMFAC) reinforcement learning that has two innovations: 1) a new objective function of training actors, composed of a policy gradient function that is related to the expected cumulative discount reward on sampled clean states and an action loss function that represents the difference between actions taken on clean and adversarial states and 2) a repetitive regularization of the action loss, ensuring the trained actors to obtain excellent performance. Furthermore, this work proposes a game model named a state-adversarial stochastic game (SASG). Despite the Nash equilibrium of SASG may not exist, adversarial perturbations to states in the RoMFAC are proven to be defensible based on SASG. Experimental results show that RoMFAC is robust against adversarial perturbations while maintaining its competitive performance in environments without perturbations.

A control system of multiple unmanned aerial vehicles (multi-UAV) is generally very complex when they complete a task in a closely-cooperative manner, e.g. two UAVs cooperatively transport a package of goods. Multi-agent reinforcement learning (MARL) offers a promising solution for such a complex control. However, MARL heavily relies on trial-and-error explorations, facing a big challenge in gathering real-world training data. Simulation environments are commonly used to overcome this challenge, i.e., a control policy is trained in a simulation environment and then transferred into a real-world system. But there often exists a gap between simulation and reality and thus a successful transfer is not guaranteed easily. The domain randomization method provides a workable way to bridge this gap. Nevertheless, the traditional one used in a policy training often suffers from slow convergence and results in an unstable decision policy. To address these issues, this paper proposes an adversarial domain randomization method. It utilizes an adversarial generator as a “nature player” to generate a more reasonable training environment so that the trained decision policy can deal with complex situations. Additionally, we improve the prioritized experience replay method by which we can sample those critical experiences, increasing the convergence speed of a training without decreasing the performance of the trained policy. We apply our method to a real-world task of dual-UAV cooperative transportation, and experiments illustrate its effectiveness compared to traditional ones.

Multiple unmanned aerial vehicles (Multi-UAV) systems have recently demonstrated significant advantages in some real-world scenarios, but the limited communication range of UAVs poses great challenges to multi-UAV collaborative decision-making. By constructing the multi-UAV cooperation problem as a multi-agent system (MAS), the cooperative decision-making among UAVs can be realized by using multi-agent reinforcement learning (MARL). Following this paradigm, this work focuses on developing partially observable MARL models that capture important information from local observations in order to select effective actions. Previous related studies employ either probability distributions or weighted mean field to update the average actions of neighborhood agents. However, they do not fully consider the feature information of surrounding neighbors, resulting in a local optimum often. In this paper, we propose a novel partially multi-agent reinforcement learning algorithm to remedy this flaw, which is based on graph attention network and partially observable mean field and is named as the GPMF algorithm for short. GPMF uses a graph attention module and a mean field module to describe how an agent is influenced by the actions of other agents at each time step. The graph attention module consists of a graph attention encoder and a differentiable attention mechanism, outputting a dynamic graph to represent the effectiveness of neighborhood agents against central agents. The mean field module approximates the effect of a neighborhood agent on a central agent as the average effect of effective neighborhood agents. Aiming at the typical task scenario of large-scale multi-UAV cooperative roundup, the proposed algorithm is evaluated based on the MAgent framework. Experimental results show that GPMF outperforms baselines including state-of-the-art partially observable mean field reinforcement learning algorithms, providing technical support for large-scale multi-UAV coordination and confrontation tasks in communication-constrained environments.

Multiagent Reinforcement Learning (MARL) plays a pivotal role in intelligent vehicle systems, offering solutions for complex decision-making, coordination, and adaptive behavior among autonomous agents. This review aims to highlight the importance of fostering trust in MARL and emphasize the significance of MARL in revolutionizing intelligent vehicle systems. First, this paper summarizes the fundamental methods of MARL. Second, it identifies the limitations of MARL in safety, robustness, generalization, and ethical constraints and outlines the corresponding research methods. Then we summarize their applications in intelligent vehicle systems. Considering human interaction is essential to practical applications of MARL in various domains, the paper also analyzes the challenges associated with MARL's applications in human-machine systems. These challenges, when overcome, could significantly enhance the real-world implementation of MARL-based intelligent vehicle systems.