13 September 2022 in IEEE Transactions on Robotics
The robotic manipulation of deformable linear objects (DLOs) has broad application prospects in many fields. However, a key issue is to obtain the exact deformation models (i.e., how robot motion affects DLO deformation), which are hard to theoretically calculate and vary among different DLOs. Thus, the shape control of DLOs is challenging, especially for large deformation control that requires global and more accurate models. In this article, we propose a coupled offline and online data-driven method for efficiently learning a global deformation model, allowing for both accurate modeling through offline learning and further updating for new DLOs via online adaptation. Specifically, the model approximated by a neural network is first trained offline on random data, then seamlessly migrated to the online phase, and further updated online during actual manipulation. Several strategies are introduced to improve the model's efficiency and generalization ability. We propose a convex-optimization-based controller and analyze the system's stability using the Lyapunov method. Detailed simulations and real-world experiments demonstrate that our method can efficiently and precisely estimate the deformation model and achieve the large deformation control of untrained DLOs in 2-D and 3-D dual-arm manipulation tasks better than the existing methods. It accomplishes all 24 tasks with different desired shapes on different DLOs in the real world, using only simulation data for the offline learning.
21 September 2024 in The International Journal of Robotics Research
Constrained environments, compared with open spaces without other objects, are more common in practical applications of manipulating deformable linear objects (DLOs) by robots, where movements of both DLOs and robot manipulators should be constrained and unintended collision should be avoided. Such a task is high-dimensional and highly constrained owing to the highly deformable DLOs, dual-arm robots with high degrees of freedom, and 3-D complex environments, which render global planning extremely challenging. Furthermore, accurate DLO models needed by planning are often unavailable owing to their strong nonlinearity and diversity, resulting in unreliable planned paths. This article focuses on the global moving and shaping of DLOs in constrained environments by dual-arm robots. The main objectives are (1) to efficiently and accurately accomplish this task and (2) to achieve generalizable and robust manipulation of various DLOs. To this end, we propose a complementary framework with whole-body planning and control using appropriate DLO model representations. First, a global planner is proposed to efficiently find feasible solutions based on a simplified DLO energy model, which considers the full system states and all constraints to plan more reliable paths. Then, a closed-loop manipulation scheme is proposed to compensate for the modeling errors and enhance the robustness and accuracy, which incorporates a constrained model predictive controller to track the planned path as guidance while real-time adjusting the robot motion based on an adaptive DLO motion model. This framework systematically considers multiple constraints for this problem, including stable deformation, overstretch prevention, closed-chain movements, and collision avoidance. The key novelty is that it can efficiently solve the high-dimensional problem subject to all those constraints and generalize to various DLOs without elaborate model identifications. Experiments demonstrate that our framework can accomplish considerably more complicated tasks than existing works. It achieves a 100% planning success rate among thousands of trials with an average time cost of less than 15 seconds, and a 100% manipulation success rate among 135 real-world tests on five different DLOs.
26 March 2025 in IEEE Robotics and Automation Letters
In-hand manipulation using multiple dexterous fingers is a critical robotic skill that can reduce the reliance on large arm motions, thereby saving space and energy. This letter focuses on in-grasp object movement, which refers to manipulating an object to a desired pose through only finger motions within a stable grasp. The key challenge lies in simultaneously achieving high precision and large-range movements while maintaining a constant stable grasp. To address this problem, we propose a simple and practical approach based on kinematic trajectory optimization with no need for pretraining or object geometries, which can be easily applied to novel objects in real-world scenarios. Adopting this approach, we won the championship for the in-hand manipulation track at the 9th Robotic Grasping and Manipulation Competition (RGMC) held at ICRA 2024. Implementation details, discussion, and further quantitative experimental results are presented in this letter, which aims to comprehensively evaluate our approach and share our key takeaways from the competition. Supplementary materials including video and code are available at https://rgmc-xl-team.github.io/ingrasp_manipulation.
1 January 2026 in IEEE Robotics and Automation Practice
17 November 2025 in IEEE Robotics and Automation Practice
In-hand manipulation has become increasingly popular in recent robotics research, probably due to the growing trend towards humanoids and Artificial General Intelligence (AGI). Although existing works show promising results, they typically require expensive dexterous hands, depth cameras and tactile sensors which are challenging for algorithm reproduction and large-scale applications. To address this issue, this paper proposes a practical and low-cost solution to the classic in-hand cube reorientation task. The proposed method is characterized by a hierarchical structure. Specifically, several in-hand motion primitives, such as object rotation and flipping, are trained using Reinforcement Learning (RL). A high-level decision module switches between these motion primitives, adapting to specific task requirements without the need for retraining. We demonstrate that the proposed method achieves continuous reorientation of the cube for nearly twenty minutes using only a low-cost LEAP Hand and a single RGB camera. The proposed method was the winning solution for in-hand manipulation at the 9th Robotic Grasping and Manipulation Competition (RGMC) during ICRA 2024. Implementation details and evaluation results are discussed in this paper. We also open-source hardware designs, code, and videos on https://rgmc-xl-team.github.io/inhand_reorientation/ to encourage re-implementation and further development.