Communication stands as a potent mechanism to harmonize the behaviors of multiple agents. However, existing work primarily concentrates on broadcast communication, which not only lacks practicality, but also leads to information redundancy. This surplus, one-fits-all information could adversely impact the communication efficiency. Furthermore, existing works often resort to basic mechanisms to integrate observed and received information, impairing the learning process. To tackle these difficulties, we propose Targeted and Trusted Multi-Agent Communication (T2MAC), a straightforward yet effective method that enables agents to learn selective engagement and evidence-driven integration. With T2MAC, agents have the capability to craft individualized messages, pinpoint ideal communication windows, and engage with reliable partners, thereby refining communication efficiency. Following the reception of messages, the agents integrate information observed and received from different sources at an evidence level. This process enables agents to collectively use evidence garnered from multiple perspectives, fostering trusted and cooperative behaviors. We evaluate our method on a diverse set of cooperative multi-agent tasks, with varying difficulties, involving different scales and ranging from Hallway, MPE to SMAC. The experiments indicate that the proposed model not only surpasses the state-of-the-art methods in terms of cooperative performance and communication efficiency, but also exhibits impressive generalization.
Due to the partial observability in decentralized multi-agent systems, communication is critical for cooperation. Furthermore, the ability to decide when and whom to communicate is important to achieve efficient communication. However, the existing methods are typically driven by extrinsic rewards. Hence, when the reward from environment is sparse, delayed, or noisy, the communication performance of these methods would be restricted. Furthermore, it would introduce additional difficulty named credit assignment when using extrinsic reward to train communication and sample policies together. To tackle these difficulties, we introduce the mechanism of intrinsic motivation from psychology to multi-agent communication. We hold the view that the observations with more uncertainty and curiosity are more valuable for communication. It can help agent find useful information from observations. It is a good complement to existing extrinsic driven methods. Concretely, at sending end, we learn a curiosity from local observations to model the communication importance. Then, we design a heuristic mechanism to prune unnecessary messages. It can solve the problem of when to communicate. Then, the ability to gate unnecessary message can reduce the cost and improve the efficiency of communication, which is important to apply to real-world scenarios. Furthermore, at receiving end, we utilize the intrinsic importance to differentiate information, which can be helpful for local decisions. It could solve the problem of whom to communicate. The ability to pay attention to useful information can efficiently improve the performance of communication behaviors. At last, we evaluate our method on a variety of multi-agent scenarios. The experiments of full communication demonstrate that the curiosity is capable to model the communication importance, and the results of gated communication further prove the conclusion.
Exploring efficiently in complex environments is still a challenging problem in reinforcement learning. Recent exploration algorithms based on "optimism in the face of uncertainty" or intrinsic motivation achieved promising performance in sparse reward settings, but they often rely on additional structures which are hard to build in large scale problems. It renders them impractical and hinders the process of combining with reinforcement learning algorithms. Hence, the most state-of-the-art RL algorithms still use the naive action space noise as exploration strategy. In this paper, we model the uncertainty about environment through agent’s ability to estimate the value across state and action space. Then, we parameterize the uncertainty by a neural network and regard it as a reward bonus signal to reward uncertain states. In this way, we generate an end-to-end bonus which can scale to complex environments with less computational cost. In order to prove the effectiveness of our method, we evaluate it on the challenging Atari 2600 games. We observed that our method achieves superior or comparable exploratory performance compared to action space noise in all environments, including environments whose rewards are sparse. The results demonstrate that our exploration method can motivate agent to explore effectively even in complex environments and it generally outperforms the naive action space noise.
Offline goal-conditioned reinforcement learning (GCRL) provides a practical framework for learning policies from fixed offline datasets to accomplish diverse goals. Existing methods commonly construct goal-conditioned learning signals through goal relabeling, reusing achieved states in offline trajectories as training goals. However, conventional relabeling methods typically treat these states uniformly, failing to prioritize informative, task-advancing goals. In this work, we propose Key-State Relabeling (KSR), a plug-and-play goal relabeling strategy that prioritizes task-relevant key states as relabeled goals. To efficiently identify key states in complex long-horizon offline trajectories, we leverage the reasoning capabilities of large language models (LLMs) to generate executable discriminators from task specifications. For trustworthy and controllable relabeling, we further introduce a two-stage Self-Check mechanism to mitigate hallucinations in LLM-generated discriminators: compile-time code validation and TD-error-guided relabel-time refinement. Theoretically, we show that KSR yields a tighter sample-complexity bound by reducing the covering complexity of the relabeled goal set and lowering the effective GOAL-BE dimension. Empirically, we show that KSR, as a plug-and-play goal relabeling strategy, consistently improves multiple offline GCRL baselines across diverse OGBench navigation and manipulation tasks, while remaining effective on noisy datasets and introducing only moderate computational overhead. Overall, key-state relabeling provides an effective way to construct more informative training transitions for goal-conditioned policy learning.