Completing complex tasks in unpredictable settings challenges robotic systems, requiring a step change in machine intelligence. Sensorimotor abilities are considered integral to human intelligence. Thus, biologically inspired machine intelligence might usefully combine artificial intelligence with robotic sensorimotor capabilities. Here we report an embodied large-language-model-enabled robot (ELLMER) framework, utilizing GPT-4 and a retrieval-augmented generation infrastructure, to enable robots to complete long-horizon tasks in unpredictable settings. The method extracts contextually relevant examples from a knowledge base, producing action plans that incorporate force and visual feedback and enabling adaptation to changing conditions. We tested ELLMER on a robot tasked with coffee making and plate decoration; these tasks consist of a sequence of sub-tasks from drawer opening to pouring, each benefiting from distinct feedback types and methods. We show that the ELLMER framework allows the robot to complete the tasks. This demonstration marks progress towards scalable, efficient and ‘intelligent robots’ able to complete complex tasks in uncertain environments.
21 January 2024 in Expert Systems with Applications
This paper develops a more general dynamics controller to generate whole-body behaviors for a quadruped-on-wheel robot. To track the quadruped centroidal motion, the wheeled motion is achieved by combining the wheel contact constraints and the centroidal momentum/dynamics model. The dynamics controller is based on a new hybrid hierarchical and prioritized weighted optimization framework. We propose one concept of a recursively updated dynamics model and this model enables to integrate the new prioritized weighted scheme in the hierarchical framework. In contrast with the conventional weighted scheme, we propose to use null-space projections among its sub-tasks. Then the prioritized impedance controller is proposed and integrated in our dynamics model, which enables to influence the hierarchical and prioritized weighted tasks in a decoupled way. The task accelerations in the two schemes are extracted with quadratic forms depending on the actuated torque and the prioritized impedance force using null-space based inverse dynamics. The inequality constraints are modified to ensure the compatibility with the hybrid convex optimization. This dynamics controller is more general and its algorithm is given completely which enables our robot to track the centroidal motion on rough terrain and handle other missions in three simulation scenarios.
Planning multi-contact motions in a receding horizon fashion requires a value function to guide the planning with respect to the future, e.g., building momentum to traverse large obstacles. Traditionally, the value function is approximated by computing trajectories in a prediction horizon (never executed) that foresees the future beyond the execution horizon. However, given the non-convex dynamics of multi-contact motions, this approach is computationally expensive. To enable online Receding Horizon Planning (RHP) of multi-contact motions, we find efficient approximations of the value function. Specifically, we propose a trajectory-based and a learning-based approach. In the former, namely RHP with Multiple Levels of Model Fidelity, we approximate the value function by computing the prediction horizon with a convex relaxed model. In the latter, namely Locally-Guided RHP, we learn an oracle to predict local objectives for locomotion tasks, and we use these local objectives to construct local value functions for guiding a short-horizon RHP. We evaluate both approaches in simulation by planning centroidal trajectories of a humanoid robot walking on moderate slopes, and on large slopes where the robot cannot maintain static balance. Our results show that locally-guided RHP achieves the best computation efficiency (95%-98.6% cycles converge online). This computation advantage enables us to demonstrate online receding horizon planning of our real-world humanoid robot Talos walking in dynamic environments that change on-the-fly.