In large-scale multi-agent systems, the existence of a large number of agents with different target tasks and connected by complex game relationships causes great difficulty for policy learning. Therefore, simplifying the learning process is an important issue. In multi-agent systems, agents with the same target tasks or attributes often interact more with each other and exhibit behaviors more similar. That means there are stronger collaborations between these agents. Most existing multi-agent reinforcement learning (MARL) algorithms expect to learn the collaborative strategies of all agents directly in order to maximize the common rewards. This causes the difficulty of policy learning to increase exponentially as the number and types of agents increase. To address this problem, we propose a goal-based hierarchical group communication (GHGC) algorithm. This algorithm divides the agents into different groups, and maintains the group’s cognitive consistency through knowledge sharing. Subsequently, we introduce a group communication and value decomposition method to ensure cooperation between the various groups. Experiments demonstrate that our model outperforms state-of-the-art MARL methods on the widely adopted StarCraft II benchmarks across different scenarios, and also possesses potential value for large-scale real-world applications.
The large action space is a key challenge in reinforcement learning. Although hierarchical methods have been proven to be effective in addressing this issue, they are not fully explored. This paper combines domain knowledge with hierarchical concepts to propose a novel Hierarchical Reinforcement Learning framework based on macro actions (HRL-MA). This framework includes a macro action mapping model that abstracts sequences of micro actions into macro actions, thereby simplifying the decision-making process. Macro actions are divided into two categories: combat macro actions (CMA) and non-combat macro actions (NO-CMA). NO-CMA are driven by decision tree-based logical rules and provide conditions for the execution of CMA. CMA form the action space of the reinforcement learning algorithm, which dynamically selects actions based on the current state. Comprehensive tests on the StarCraft II maps Simple64 and AbyssalReefLE demonstrate that the HRL-MA framework exhibits superior performance, achieving higher win rates compared to baseline algorithms. Furthermore, in mini-game scenarios, HRL-MA consistently outperforms baseline algorithms in terms of reward scores. The findings highlight the effectiveness of integrating hierarchical structures and macro actions in reinforcement learning to manage complex decision-making tasks in environments with large action spaces.