We have extended the task-space multiobjective controllers that write as quadratic programs (QPs) to handle multirobot systems as a single centralized control. The idea is to assemble all the “robots” models and their interaction task constraints into a single QP formulation. By multirobot, we mean that whatever entities a given robot will interact with (solid or articulated systems, actuated, partially or not at all, fixed-base or floating-base), we model them as clusters of robots and the controller computes the state of each cluster as an overall system and their interaction forces in a physically consistent way. By doing this, the tasks specification simplifies substantially. At the heart of the interactions between the systems are the contact forces; methodologies are provided to achieve reliable force tracking by our multirobot QP controller. The approach is assessed by a large panel of experiments on real complex robotic platforms (full-size humanoid, dexterous robotic hand, fixed-base anthropomorphic arm) performing whole-body manipulations, dexterous manipulations, and robot–robot comanipulations of rigid floating objects and articulated mechanisms, such as doors, drawers, boxes, or even smaller mechanisms like a spring-loaded click pen.
31 October 2019 in IEEE Robotics & Automation Magazine
We consider the problem of estimating realistic contact forces during manipulation, backed with ground-truth measurements, using vision alone. Interaction forces are usually measured by mounting force transducers onto the manipulated objects or the hands. Those are costly, cumbersome, and alter the objects' physical properties and their perception by the human sense of touch. Our work establishes that interaction forces can be estimated in a cost-effective, reliable, non-intrusive way using vision. This is a complex and challenging problem. Indeed, in multi-contact, a given motion can generally be caused by an infinity of possible force distributions. To alleviate the limitations of traditional models based on inverse optimization, we collect and release the first large-scale dataset on manipulation kinodynamics as 3.2 hours of synchronized force and motion measurements under 193 object-grasp configurations. We learn a mapping between high-level kinematic features based on the equations of motion and the underlying manipulation forces using recurrent neural networks (RNN). The RNN predictions are consistently refined using physics-based optimization through second-order cone programming (SOCP). We show that our method can successfully capture interaction forces compatible with both the observations and the way humans intuitively manipulate objects, using a single RGB-D camera.
13 September 2017 in IEEE Transactions on Automatic Control
We propose a formal analysis with some theoretical properties of weight-prioritized multi-task inverse-dynamics-like control of humanoid robots, being a case of redundant “manipulators” with a non-actuated free-floating base and multiple unilateral frictional contacts with the environment. The controller builds on a weighted sum scalarization of a multiobjective optimization problem under equality and inequality constraints, which appears as a straightforward solution to account for state and control input viability constraints characteristic of humanoid robots that were usually absent from early existing pseudo-inverse and null-space projection-based prioritized multi-task approaches. We argue that our formulation is indeed well founded and justified from a theoretical standpoint, and we propose an analysis of some stability properties of the approach: Lyapunov stability is demonstrated for the closed-loop dynamical system that we analytically derive in the unconstrained multiobjective optimization case. Stability in terms of solution existence, uniqueness, continuity, and robustness to perturbations, is then formally demonstrated for the constrained quadratic program.
26 March 2026 in IEEE/ASME Transactions on Mechatronics
Humanoid robots hold great potential to perform various human-level skills, involving unified locomotion and manipulation in real-world settings. Driven by advances in machine learning and the strength of existing model-based approaches, these capabilities have progressed rapidly, but often separately. This survey offers a comprehensive overview of the state-of-the-art in humanoid locomotion and manipulation, with a focus on control, planning, and learning methods. We first review the model-based methods that have been the backbone of humanoid robotics for the past three decades. We discuss contact planning, motion planning, and whole-body control, highlighting the tradeoffs between model fidelity and computational efficiency. Then, the focus is shifted to examine emerging learning-based methods, with an emphasis on reinforcement and imitation learning that enhance the robustness and versatility of loco-manipulation skills. Furthermore, we assess the potential of integrating foundation models with humanoid embodiments to enable the development of generalist humanoid agents. This survey also highlights the emerging role of tactile sensing, particularly whole-body tactile feedback, as a crucial modality for handling contact-rich interactions. Finally, we compare the strengths and limitations of model-based and learning-based paradigms from multiple perspectives, such as robustness, computational efficiency, versatility, and generalizability, and suggest potential solutions to existing challenges.