It is important for humanoid-like mobile robots to learn the complex motion sequences in human-robot environment such that the robots can adapt such motions. This paper describes a reinforcement learning (RL) strategy for manipulation and grasping of a mobile manipulator, which reduces the complexity of the visual feedback and handle varying manipulation dynamics and uncertain external perturbations. Two hierarchies plannings have been considered in the proposed strategy, high-level online redundancy resolution based on the neural-dynamic optimization algorithm in operational space, and low-level reinforcement learning in joint space. At this level, the dynamic movement primitives (DMPs) have been considered to model and learn the joint trajectories, and then the reinforcement learning is employed to learn the trajectories with uncertainties. Experimental results on the developed humanoid-like mobile robot demonstrate that the presented approach can suppress the uncertain external perturbations.

In human and robot collaborative hybrid assembly cell as we proposed, it is important to develop automatic subtask allocation strategy for human and robot in usage of their advantages. We introduce a folk-joint task model that describes the sequential and parallel features and logic restriction of human and robot collaboration appropriately. To preserve a cost-effectiveness level of task allocation, we develop a logic mathematic method to quantitatively describe this discrete-event system by considering the system tradeoff between the assembly time cost and payment cost. A genetic based revolutionary algorithm is developed for real-time and reliable subtask allocation to meet the required cost-effectiveness. This task allocation strategy is built for a human worker and collaborates with various robot co-workers to meet the small production situation in future. The performance of proposed algorithm is experimentally studied, and the cost-effectiveness is analyzed comparatively on an electronic assembly case.

This letter describes an approach to achieve well-known Chinese cooking art stir-fry on a bimanual robot system. Stir-fry requires a sequence of highly dynamic coordinated movements, which is usually difficult to learn for a chef, let alone transfer to robots. In this letter, we define a canonical stir-fry movement, and then propose a decoupled framework for learning this deformable object manipulation from human demonstration. First, dualarms of the robot are decoupled into different roles (a leader and follower) and learned with classical and neural network based methods separately, then the bimanual task is transformed into a coordination problem. To obtain general bimanual coordination, we secondly propose a Graph and Transformer based model— Structured-Transformer , to capture the spatio-temporal relationship between dual-arm movements. Finally, by adding visual feedback of contents deformation, our framework can adjust the movements automatically to achieve the desired stir-fry effect. We verify the framework by a simulator and deploy it on a real bimanual Panda robot system. The experimental results validate our framework can realize the bimanual robot stir-fry motion and have the potential to extend to other deformable objects with bimanual coordination.

Industry 4.0 demands the heavy usage of robotic mobile manipulators with high autonomy and intelligence. The goal is to accomplish dexterous manipulation tasks without prior knowledge of the object status in unstructured environments. It is important for the mobile manipulator to recognize and detect the objects, determine manipulation pose and adjust its pose in the workspace fast and accurately. In this research, we developed a stereo vision algorithm for the object pose estimation using point cloud data from multiple stereo vision systems. An improved iterative closest point algorithm and method is developed for the pose estimation. With the pose input, algorithms and several criteria are studied for the robot to select and adjust its pose by maximizing its manipulability on a given manipulation task. The performance of each technical modules and the complete robotic system is finally shown by virtual robot in the simulator and real robot in experiments.

Human walking intention estimation is a critical step for the active assistance control of lower limb exoskeleton, because the purpose of active assistance control is human motion assistance rather than human motion tracking. Complying with human walking intention is the basic requirement of human walking assistance. Hence, the human walking intention must be estimated first to ensure exoskeleton will not impede human motion. Actually, estimating human walking intention is to estimate human joint torque during walking. In order to estimate a smooth personalized human joint torque profile, an online learning and prediction algorithm of human joint trajectory and joint torque is proposed in this paper. The algorithm is based on dynamical movement primitives model which is used for online learning and predicting human joint trajectory which is substituted into the human dynamics model to estimate the human joint torque. The results of human walking experiments demonstrate that the proposed algorithm can not only predict a smooth human joint trajectory and joint torque profile in real time, but also compensate the phase delay caused by sensor signal filtering. Hence, the proposed algorithm is suitable for the active walking assistance control of exoskeleton.

Malicious use of micro aerial vehicles (MAVs) has become a serious threat to public safety and personal privacy in recent years. Motivated by this problem, we propose a systematic approach to monitor the intrusion of malicious MAVs based on a novel type of panoramic stereo camera networks. Each sensing node of such a network consists of 16 lenses that can form a 360-degree panoramic vision system. The 16 lenses further form 8 pairs of stereo cameras that can directly localize aerial targets. The effective range for a sensing node localizing a MAV like DJI M300 could reach 80 meters, which is much farther than existing commercial stereo cameras. In terms of algorithms, we propose i) a novel visual MAV detection algorithm based primarily on motion features of MAVs, ii) an efficient stereo localization algorithm based on sparse feature points, and iii) robust multi-target tracking and trajectory fusion algorithm to fuse the observations of different sensing nodes. The effectiveness, robustness, and accuracy of the proposed algorithms together with the overall system have been verified by extensive experimental tests. To the best of our knowledge, this is the first systematic approach to detect, localize, and track unknown MAVs in the literature. Our approach provides a scalable solution to securely cover large areas of interest against malicious MAV intrusion.