28 February 2024 in IEEE Robotics and Automation Letters
To complete tasks in dynamic environments, robots need to timely update their plans to react to environment changes. Traditional stripe-like or learning-based planners struggle to achieve this due to their high reliance on meticulously predefined planning rules or labeled data. Fortunately, recent works find that Large Language Models (LLMs) can be effectively prompted to solve planning problems. Thus, we investigate the strategies for LLMs to master reactive planning problems without complex definitions and extra training. We propose Text2Reaction, an LLM-based framework enabling robots to continuously reason and update plans according to the latest environment changes. Inspired from human's step-by-step re-planning process, we present the Re-planning Prompt, which informs LLMs the basic principles of re-planning and fosters the gradual development of a current plan to a new one in a three-hop reasoning manner–cause analysis, consequence inference, and plan adjustment. In addition, Text2Reaction is designed to first generate an initial plan based on the task description before execution, allowing for subsequent iterative updates of this plan. We demonstrate the superior performance of Text2Reaction over prior works in reacting to various environment changes and completing varied tasks. In addition, we validate the reliability of our re-planning prompt through ablation experiments and its capability when deployed in real-world robots, enabling continuous reasoning in the face of diverse changes until the user instructions are successfully completed.
12 May 2023 in IEEE Transactions on Instrumentation and Measurement
Biomimetic underwater vehicles (BUVs) are important instrumentation in applications such as water quality monitoring and underwater detection. In this paper, we study the cooperative trajectory planning problem for multiple BUVs and propose a stepwise trajectory planning algorithm based on temporal-spatial Bezier curves to enable cooperative formations. We introduce the time variable into the three-dimensional (3D) Bezier curve, designing a four-dimensional temporal-spatial Bezier curve. Considering the various physical constraints of the BUV, we obtain a smooth and feasible trajectory using Particle Swarm Optimization (PSO) when taking the shortest trajectory distance as the optimization goal. On this basis, we propose a method to perform cooperative trajectory planning and efficient obstacle avoidance replanning for multiple BUVs by integrating the real-time collision avoidance, real-time detection, and underwater complex terrain constraints. We verify the effectiveness of the proposed cooperative planning method through simulation and comparison.
5 June 2025 in IEEE Transactions on Robotics
Visuotactile sensors have been shown to provide rich contact information for robots. However, how to build a high-fidelity visuotactile simulator that supports multi-mode tactile imprints and various sensor configurations (such as coating patterns) remains a challenging problem. In this paper, we present TacFlex, an efficient and flexible simulator for visuotactile sensors, which physically simulates the elastomer deformation using Finite Element Methods (FEM), and focuses on linking the deformed elastomer mesh to diverse tactile imprints, including tactile images with arbitrary coating patterns and tactile 3D point clouds. We further propose a ray tracing-based rectification method to deal with multi-medium refraction effects to make the simulated tactile images more realistic. Extensive qualitative and quantitative experiments are conducted to demonstrate the effectiveness of TacFlex on several visuotactile sensors. Furthermore, we explore the Sim2Real performance of different tactile imprints provided by TacFlex in tactile perception and manipulation tasks, such as cylindrical object pose estimation and peg-in-hole. The perception/policy models trained in simulation are successfully deployed in the real world. Finally, we present the outlook on the potential of TacFlex in visuotactile manipulation learning. The TacFlex simulator is open-sourced to the community. See supplementary video, code, and results at https://sites.google.com/view/tacflex/.
21 April 2025 in IEEE Robotics and Automation Letters
Large language models (LLMs) have demonstrated remarkable reasoning capabilities. However, in real-world robotic tasks, LLMs face grounding issues and lack precise feedback, resulting in the generated solutions deviating from the actual situation. In this paper, we propose Double-Feedback, a method that enhances LLMs reasoning by Knowledge graphs (KGs). The KGs play three key roles in Double-Feedback: prompting the LLMs to generate solutions, representing the task scenes, and verifying the solutions to provide feedback. We design structured knowledge prompts that convey the task knowledge background, example solutions, revision principles, and robotic tasks to the LLMs. We also introduce the distributed representation to quantify the task scene with interpretability. Based on the structured knowledge prompts and the distributed representation, we employ the KGs to evaluate the feasibility of each step before execution and verify the effects of the solutions after completing the tasks. The LLMs can adjust and replan the solutions based on the feedback from the KGs. Extensive experiments demonstrate that Double-Feedback outperforms prior works in the ALFRED benchmark. In addition, ablation studies show that Double-Feedback guides LLMs in generating solutions aligned with robotic tasks in the real world.
1 January 2026 in IEEE Transactions on Industrial Electronics
1 January 2026 in IEEE Transactions on Industrial Electronics