19 March 2026 in Expert Systems with Applications
12 November 2025 in IEEE Transactions on Automation Science and Engineering
Learning multimodal policies is crucial for enhancing exploration in online reinforcement learning (RL), especially in tasks with continuous action spaces and non-convex reward landscapes. While recent diffusion policies show promise, they often suffer from low computational efficiency in online settings. A more training-efficient paradigm involves modeling the policy as a Boltzmann distribution and guiding the action sampling directly with the gradient of the Q-value with respect to the action (proportional to the score function of the policy), such as via Langevin dynamics. However, analysis in this paper reveals that this gradient-guided approach suffers from two critical challenges: sampling instability caused by the widely varying magnitude of action gradients; and mode imbalance, where the sampling process inaccurately represents the weights of different high-value action modes. To address these challenges, this paper introduces three targeted techniques: score normalization and reshaping to stabilize the sampling process, and value-based resampling to correct mode imbalance. These techniques are then integrated into an actor-critic framework, resulting in the Score-Enhanced Actor-Critic (SEAC) algorithm. Simulation and real-world experiments demonstrate that SEAC not only effectively learns multimodal behaviors but also achieves state-of-the-art performance and high computational efficiency compared to prior multimodal RL methods. The code of this paper is available at https://github.com/THUzouwenjun/SEAC.
15 May 2026 in IEEE/ASME Transactions on Mechatronics
Inverse kinematics (IK) determines the joint configurations needed to achieve a specific end pose of a kinematic chain. IK solutions for complex robot manipulators are typically nonunique, allowing multiple joint configurations to correspond to the same end pose. However, traditional IK methods generally yield only a single solution. To overcome this limitation, this study introduces the diffusion inverse kinematics (DIK) method, which can provide multiple IK solutions for the same end pose. This capability allows for the selection of the optimal configuration based on criteria, such as accuracy, proximity to the current position, or adherence to specific constraints. DIK utilizes a diffusion generation process akin to the sampling process of the diffusion model to generate these multimodal solutions, and it employs a self-supervised learning framework to align the training data with real-world applications. This framework updates the parameters of the diffusion generation process by minimizing the difference between the actual poses and the target poses. For robot manipulators that operate under specific constraints, DIK integrates these constraints into the IK solution process, and addresses them using exterior point methods. Experimental results affirm that DIK not only generates multimodal IK solutions but also achieves high accuracy and zero-collision outcomes.