28 August 2020 in IEEE Transactions on Industrial Informatics
Multiple peg-in-hole insertion is one of the challenging tasks in precision assembly. In this work, an insertion policy learning method is proposed for multiple peg-in-hole precision assembly. The insertion policy learning is separated into two phases: initial policy learning and residual policy learning. In initial policy learning, a state-to-action policy mapping model based on Gaussian mixture model is established. And Gaussian mixture regression is used to generalize the policy reuse. In residual policy learning, a reinforcement learning method named normalized advantage function is employed to refine the insertion policy. An adaptive action exploration is designed to improve the performance of exploration, and the prioritized experience replay is introduced to make residual policy learning from historical experience more efficient. Besides, the hierarchical reward is designed considering the contact dynamics as well as the efficiency and safety of precision insertion. Finally, experiments are conducted to validate the effectiveness of the proposed method.
3 April 2019 in IEEE/ASME Transactions on Mechatronics
This paper proposes a skill learning approach for precision assembly robot, aiming to realize efficient skill transfer from teacher to robot through several demonstrations. The framework is designed considering that a skill has multiple controllers and procedures. A complex skill is segmented to an action sequence according to the changes of the teacher's selective attention settings on the multiple system variables. The learning of each action is to select a predefined action class and learn its key parameters from the demonstration data. The action sequence forms a finite state machine. To execute an action, firstly the action instance is generated from the action class and the learned parameters. Then at each time step the action state is updated by the Gaussian mixture model based dynamical system and is sent to the lower level controller as the reference signal, so that the action state evolves towards the target with a specified motion pattern. In this work, the action classes of image feature guided motion and the force constrained motion are proposed based on the multi-camera microscopic vision and 3-dimensional force feedback, respectively, which can be reused in different skills. The proposed approach was validated by the two experiments of the sleeve-cavity assembly and the coil-cylinder assembly.
9 July 2021 in IEEE Robotics and Automation Letters
Deep learning-based methods have achieved promising results on surgical instrument segmentation. However, the high computation cost may limit the application of deep models to time-sensitive tasks such as online surgical video analysis for robotic-assisted surgery. Moreover, current methods may still suffer from challenging conditions in surgical images such as various lighting conditions and the presence of blood. We propose a novel Multi-frame Feature Aggregation (MFFA) module to aggregate video frame features temporally and spatially in a recurrent mode. By distributing the computation load of deep feature extraction over sequential frames, we can use a lightweight encoder to reduce the computation costs at each time step. Moreover, public surgical videos usually are not labeled frame by frame, so we develop a method that can randomly synthesize a surgical frame sequence from a single labeled frame to assist network training. We demonstrate that our approach achieves superior performance to corresponding deeper segmentation models on two public surgery datasets.
16 March 2023 in IEEE/ASME Transactions on Mechatronics
Biomedical microelectrode implantation is a promising approach to achieve invasive brain-machine interface with high bandwidth and good biocompatibility. However, it is difficult to manipulate a microelectrode probe because of its micron-level width and flexible material. Toward the challenging robotic implantation, this article addresses an essential operation step: automated hooking of flexible microelectrode probe with a micro-needle. First, a point-contour of interest extraction network for microscopic image (MicroPCIE) is proposed to obtain key image features for visual guidance. Based on few-shot learning mechanism, MicroPCIE trained on nonspecific dataset can be rapidly deployed to the specific hooking task without requiring data-driven optimization. Second, a robotic hooking control system is designed considering the distributed probe locations, variable needle direction, and efficiency requirements. The needle-probe alignment control is executed under low and high microscope magnifications, to realize larger operation range and permit high alignment precision. The optical design and camera aiming control are proposed to guarantee the visibility and clearness of objects after low-to-high magnification switch. The effectiveness of the proposed methods is validated with a series of experiments.
26 June 2023 in IEEE Transactions on Instrumentation and Measurement
Image-based visual servoing methods have been widely used in many areas including the position alignment for assembly. However, the parallel vision system suffers from low sensitivity to motion along the optical axis due to the camera’s perspective characteristics, which can lead to a larger alignment error along the optical axis of the camera and affect subsequent tasks. To address this issue, an innovative binocular Eye-to-Hand (ETH) vision system with two cameras whose optical axes are approximate orthogonal is designed. By deriving and analyzing the interaction matrix of point feature for the proposed system, its ability is demonstrated to improve sensitivity for motion along the optical axis. Furthermore, an image-based visual servoing control method with the proposed ETH vision system is developed for position alignment in a screw grasping experiment. A calibration method for the image Jacobian matrix is developed based on active motions of the robot’s end-effector. Finally, the experimental results verify the effectiveness of the proposed methods. The position alignment errors in the experiments with the proposed vision system and method are less than 0.5 mm. The results indicate that the proposed method holds great potential for industrial applications in the area of small components assembly.