13 June 2024 in IEEE Access
Interactive visual navigation tasks, which involve following instructions to reach and interact with specific targets, are challenging not only because successful experiences are very rare but also because complex visual inputs require a substantial number of samples. Previous methods for these tasks often rely on intricately designed dense rewards or the use of expensive expert data for imitation learning. To tackle these challenges, we propose a novel approach, Visual Hindsight Self-Imitation Learning (VHS), which enables re-labeling in vision-based and partially observable environments through Prototypical Goal (PG) embedding. We introduce the PG embeddings, which are derived from experienced goal observations, as opposed to handling instructions as word embeddings. This embedding technique allows the agent to visually reinterpret its unsuccessful attempts, enabling vision-based goal re-labeling and self-imitation from enhanced successful experiences. Experimental results show that VHS outperforms existing techniques in interactive visual navigation tasks, confirming its superior performance, sample efficiency, and generalization.
11 December 2025 in IEEE Transactions on Cognitive and Developmental Systems
Reinforcement learning (RL) agents face fundamental challenges in balancing exploration and exploitation, particularly when sparse or dense rewards bias learning toward sub–optimal behaviors. Biological systems, notably human toddlers, naturally navigate this balance through developmental transitions from free exploration to goal-directed behavior. Inspired by this progression, we propose the Sparse-to-Dense (S2D) reward transition for goal-oriented RL tasks. Our approach transitions from sparse rewards encouraging broad exploration to dense rewards guiding efficient exploitation, while preserving optimal policies via potential-based reward shaping throughout. Through extensive experiments on robotic manipulation and egocentric 3D navigation tasks, we demonstrate that S2D transitions significantly enhance sample efficiency, and generalization compared to static reward schemes and intrinsic motivation baselines. Using a novel Cross-Density Visualizer, we reveal that S2D transitions smooth the policy loss landscape, resulting in wider minima associated with improved generalization. Furthermore, reinterpreting Tolman’s maze experiments in modern RL contexts, we show that early sparse-reward exploration establishes robust initial parameters analogous to cognitive maps, facilitating stable learning under subsequent dense rewards. Our findings bridge developmental psychology and machine learning, offering a principled framework for designing adaptive reward structures in complex RL environments.
3 February 2026 in IEEE Access
Applying multi-agent reinforcement learning (MARL) to real-world scenarios is challenging because agents often need to adapt quickly to unexpected situations, including those rarely or never encountered in training. Recent methods for out-of-distribution generalization are unsuitable for applications on out-of-distribution tasks with limited communication, because they are typically restricted to centralized training or some specialized instances of distribution shifts. To address this limitation, we introduce the Unexpectedness Encoding Scheme, a new decentralized MARL algorithm in which agents communicate ‘‘unexpectedness,’’ the surprising aspects of the environment. In addition to sending their usual reward-driven messages, each agent predicts the next observation based on past experience and then compares this prediction with the actual outcome. The discrepancy between the two is encoded as a message, enabling agents to adapt more effectively to sudden or extreme changes. Experimental results on multi-agent cooperative tasks demonstrate that our method adapts robustly to both dynamically changing training environments and previously unseen out-of-distribution scenarios.