Multi-agent reinforcement learning is widely used and can successfully solve many problems in the real world. In the multi-agent reinforcement learning system, a global critic network is used to guide each agent’s strategy to update to learn the most beneficial strategy for the collective. However, the global critic network also makes the current agent’s learning be affected by other agents’ strategies, which leads to unstable learning. To solve this problem, we propose Dynamic Decomposed Multi-Agent Deep Deterministic Policy Gradient (DD-MADDPG): a new network that considers both global and local evaluations and adaptively adjusts the agent’s attention to the two evaluations. Besides, the use of the experience replay buffer by Multi-Agent Deep Deterministic Policy Gradient(MADDPG) produces outdated experience, and the outdated strategies of other agents further affect the learning of the current agent. To reduce the influence of other agents’ outdated experience, we propose TDError and Time-based experience sampling (T2-PER) based on DD-MADDPG. We evaluate the proposed algorithm’s performance according to the learning stability and the average return obtained by the agents. We have conducted experiments in the MPE environment. The results show that the proposed method has better stability and higher learning efficiency than MADDPG and has a certain generalization ability.
Multi-Agent Reinforcement Learning (MARL) is widely used to solve various real-world problems. In MARL, the environment contains multiple agents. A good grasp of the environment can guide agents to learn cooperative strategies. In Centralized Training Decentralized Execution (CTDE), a centralized critic is used to guide cooperative strategies learning. However, having multiple agents in the environment leads to the curse of dimensionality and influence of other agents’ strategies, resulting in difficulties for centralized critics to learn good cooperative strategies. We propose a graph-based approach to overcome the above problems. It uses a graph neural network, which uses partial observations of agents as input, and information between agents is aggregated by graph methods to extract information about the whole environment. In this way, agents can improve their understanding of the overall state of the environment and other agents in the environment while avoiding dimensional explosion. Then we combine a dual critic dynamic decomposition method with soft actor-critic to train policy. The former uses individual and global rewards for learning, avoiding the influence of other agents’ strategies, and the latter help to learn an optional policy better. We call this approach Multi-Agent Graph-based soft Actor-Critic (MAGAC). We compare our proposed method with several classical MARL algorithms under the Multi-agent Particle Environment (MPE). The experimental results show that our method can achieve a faster learning speed while learning better policy.
9 November 2016 in Advances in Intelligent Systems and Computing
The Bulk Synchronous Parallel(BSP) computational model has emerged as the dominant distributed framework to build large-scale iterative graph processing systems. While its implementations(e.g., Pregel, Giraph, and Hama) achieve high scalability, frequent synchronization and communication among the workers can cause substantial parallel inefficiency. To help address this critical concern, this paper introduces the GraphHP(Graph Hybrid Processing) platform which inherits the friendly vertex-centric BSP programming interface and optimizes its synchronization and communication overhead. To achieve the goal, we first propose a hybrid execution model which differentiates between the computations within a graph partition and across the partitions, and decouples the computations within a partition from distributed synchronization and communication. By implementing the computations within a partition by pseudo-superstep iteration in memory, the hybrid execution model can effectively reduce synchronization and communication overhead while not requiring heavy scheduling overhead or graph-centric sequential algorithms. We then demonstrate how the hybrid execution model can be easily implemented within the BSP abstraction to preserve its simple programming interface. Finally, we evaluate our implementation of the GraphHP platform on classical BSP applications and show that it performs significantly better than the state-of-the-art BSP implementations. Our GraphHP implementation is based on Hama, but can easily generalize to other BSP platforms.