This paper proposes an improved disturbance observer to realize accurate contact force estimation using the joint torque sensor. The joint torque sensor separates the dynamics of the link side from the motor side of the robot manipulator. Therefore, only computing the partial dynamics on the link side can realize external force estimation. This can considerably reduce the modeling workload and error terms that may affect the estimation results. Furthermore, this paper presents that the observed residual value during free motion can be considered as the friction dynamics, which is approximated by the neural network (NN) due to its inherent capacity in approximating nonlinear functions. After that, the estimation accuracy of the observer is considerably improved. Compared to other local NN approximation method, we analyzes the properties of the friction force in detail to select appropriate excitation trajectory for accurate global approximation results using only limited training data. We have presented that the suitable excitation trajectory and the use of global basis function are the sufficient conditions for global friction approximation. The proof of this theorem is also given. The Kalman filter is also utilized to reduce the noise of the estimation results in real-time. The experimental results also demonstrate the efficacy of the proposed method, which accurately estimates the contact force for flexible joint robots.

In this brief, impedance adaptation is investigated for robots interacting with unknown environments, subject to unknown environment dynamics and position parameters. A cost function that measures the tracking error and interaction force is defined, and a complete state-space function considering the desired trajectory, environment dynamics, and position parameters is presented. The unknown environment dynamics and the unobservable environment position lead to unknown part of the system function as well as the system states in the complete system model. To this end, the output feedback adaptive dynamic programming (OPFB ADP) method is selected to realize the optimized impedance adaptation. Moreover, when considering the environment position/trajectory, the optimal impedance solution is difficult to obtain due to the existence of the arbitrary endpoint. An adaptive dynamic programming algorithm considering the trajectory tracking problem with arbitrary endpoint is proposed to deal with the problem. The convergence speed is accelerated by adding a discount factor. The validity of the proposed method is verified through simulation and experimental studies.

This paper proposes a novel hierarchical reinforcement learning (HRL) framework of complex manipulation tasks which integrates the human prior knowledge. The framework involves the following steps: $ \textbf{1)} $ The manipulation process is divided into several stages based on human prior knowledge. $ \textbf{2)} $ Transition conditions between stages are determined in the form of “if-then” rules. $ \textbf{3)} $ The key features of each stage are selected, and the corresponding control policies are designed via human knowledge. $ \textbf{4)} $ The policy gradient with parameter-based exploration (PGPE) method is employed to optimize the policy parameters because it does not require the policy to be derivable for the parameters. To increase the convergence speed of this framework, importance sampling and adaptive adjustment of exploration variance are employed to improve it. $ \textbf{5)} $ On this basis, to facilitate the transfer of simulation results to practical experiments, half sim-to-real method is presented, which fully utilizes the simulation results, and the differences between simulation and experimental environments are considered. Simulation and experimental studies show that our framework can deal with the peg-hole-insertion task with a high quality in less than 1600 episodes and can safely adapt the skill into the practical scene with little iterations, which verify the efficiency of the presented method. Note to Practitioners —Intelligent robots will become the right assistants of human beings in the future, especially in various areas of complex manipulation occasions. The important premise is that the robots should have certain ability of complex manipulation skill learning. Complex manipulation tasks can be decomposed into multiple stages, and HRL is a suitable and efficient method for solving this kind of problems. This paper proposes a novel HRL framework which can better integrate the human prior knowledge. In addition, improved PGPE method is proposed to obtain the optimized policy parameters more quickly. More importantly, a novel half sim-to-real transfer method is presented to better integrate the simulation and experiment results. This provides a common paradigm, which leverages the simulation results to reduce the interaction between robot and practical environment and then utilizes the experiment results to further optimize the policy parameters.

In this paper, a novel interaction control is presented to resolve the optimized robot-environment interaction control problems subject to flexible environment with unknown dynamics parameters. A cost function measuring the trajectory tracking error as well as the non-inertial interaction force is defined. A complete state space equation considering the robot desired trajectory, object dynamics and position parameters is also presented to address the optimized robot-environment interaction control problem. The improved Q-learning method is developed as the fundamental of the proposed control to deal with the challenges brought by the unknown environment dynamics and the reference position of the robot desired trajectory. Simulation and experimental studies verify the validity of the presented method.

Nowadays the robot manipulation skills are usually learned by human demonstration via trajectory-level learning, which somewhat lacks robustness and generalization. In this paper, we propose a novel contact state level learning method for robot manipulation skill acquisition via human demonstration. The robot-environment contact states are described via environment dynamics modelling and geometric constraints modelling for flexible contact and rigid contact cases, respectively. During human demonstration process, the robot-environment interaction force, the robot position, and velocity data are collected. After that, the environment dynamics and geometric constraints modelling methods are presented to determine the contact state changes during the robot manipulation process. Then the robot manipulator learns the contact state information rather than specific manipulation trajectory. On this basis, the manipulation control law using active exploration method is presented to control the robot during the button pressing process and peg-hole-insertion process, respectively. Finally, the performance of the presented methodology has been verified via experimental studies. Note to Practitioners—Intelligent robots will become the right assistants of human beings in the future, especially in various areas of manipulation occasions. The important premise of realizing this vision is that the robots should have certain ability of manipulation skill learning. A lot of research has been carried out in this field, many of which are focusing on trajectory level manipulation skill learning and reproduction. Other than the trajectory level learning, human beings can learn many other higher levels of manipulation skills, such as the contact state level and semantic level learning, which makes the learning results more robust and general. In this paper, the contact state estimation and learning method via environment dynamics and geometric constraints modelling is presented to learn the robot manipulation skill based on the contact state transition conditions. In this way, the robot needs less data in the skill learning process, and the trajectory level learning is avoided. After learning the contact state level manipulation skill, the lower trajectory level command is autonomously generated. Experiments on button pressing and peg-hole-insertion tasks by KUKA iiwa robot have obtained very good results. Other than the button pressing and peg-hole-insertion tasks, the presented methodology can be applied to many other manipulation tasks, as long as there are contact state changes in the manipulation process. The work of this paper lays a foundation for the robot learning of higher-level manipulation skills.

In this article, a novel stochastic optimal control method is developed for robot manipulator interacting with a time-varying uncertain environment. The unknown environment model is described as a nonlinear system with time-varying parameters as well as stochastic information, which is learned via the Gaussian process regression (GPR) method as the external dynamics. Integrating the learned external dynamics as well as the stochastic uncertainties, the complete interaction system dynamics are obtained. Then the iterative linear quadratic Gaussian with learned external dynamics (ILQG-LEDs) method is presented to obtain the optimal manipulation control parameters, namely, the feedforward force, the reference trajectory, as well as the impedance parameters, subject to time-varying environment dynamics. The comparative simulation studies verify the advantages of the presented method, and the experimental studies of the peg-hole-insertion task prove that this method can deal with complex manipulation tasks.