Soft robots, owing to their elastomeric material, ensure safe interaction with their surroundings. These robot compliance properties inevitably impose a trade-off against precise motion control, as to which conventional model-based methods were proposed to approximate the robot kinematics. However, too many parameters, regarding robot deformation and external disturbance, are difficult to obtain, even if possible, which could be very nonlinear. Sensors self-contained in the robot are required to compensate modelling uncertainties and external disturbances. Camera (eye) integrated at the robot end-effector (hand) is a common setting. To this end, we propose an eye-in-hand visual servo that incorporates with learning-based controller to accomplish more precise robotic tasks. Local Gaussian process regression (GPR) is used to initialize and refine the inverse mappings online, without prior knowledge of robot and camera parameters. Experimental validation is also conducted to demonstrate the hyper-elastic low-stiffness robot can compensate an external variable loading during trajectory tracking.

In the feature/object tracking of eye-in-hand visual servoing, 2D motion estimation relying only on image plane feedback is easily affected by vision occlusion, blurring, or poor lighting. For the commonly-used template matching method, tracking performance greatly depends on the image quality. Fiber Bragg gratings (FBGs), a type of high-frequency flexible strain sensor, can be used as an assistant device for soft robot control. We propose a method to enhance motion estimation in soft robotic visual servoing by fusing the results from template matching and FBG wavelength shifts to achieve more accurate tracking in applications such as minimally invasive surgery. Path following performance is validated in a simulated laparoscopic scene and LEGO®-constructed scene, demonstrating significant improvement to feature tracking and robot motion, even under external forces.

Cardiac electrophysiology is an effective treatment for atrial fibrillation, in which a long, steerable catheter is inserted into the heart chamber to conduct radio frequency ablation. Magnetic resonance imaging (MRI) can provide enhanced intraoperative monitoring of the ablation progress as well as the localization of catheter position. However, accurate and real-time tracking of the catheter shape and its efficient manipulation under MRI remains challenging. In this article, we designed a shape tracking system that integrates a multicore fiber Bragg grating (FBG) fiber and tracking coils with a standard cardiac catheter. Both the shape and positional tracking of the bendable section could be achieved. A learning-based modeling method is developed for cardiac catheters, which uses FBG-reconstructed three-dimensional curvatures for model initialization. The proposed modeling method was implemented on an MRI-guided robotic platform to achieve feedback control of a cardiac catheter. The shape tracking performance was experimentally verified, demonstrating 2.33° average error for each sensing segment and 1.53 mm positional accuracy at the catheter tip. The feedback control performance was tested by autonomous targeting and path following (average deviation of 0.62 mm) tasks. The overall performance of the integrated robotic system was validated by a pulmonary vein isolation simulator with ex-vivo tissue ablation, which employed a left atrial phantom with pulsatile liquid flow. Catheter tracking and feedback control tests were conducted in an MRI scanner, demonstrating the capability of the proposed system under MRI.

Image processing has significantly extended the practical value of the eye-in-hand camera, enabling and promoting its applications for quantitative measurement. However, fully vision-based pose estimation methods sometimes encounter difficulties in handling cases with deficient features. In this article, we fuse visual information with the sparse strain data collected from a single-core fiber inscribed with fiber Bragg gratings (FBGs) to facilitate continuum robot pose estimation. An improved extreme learning machine algorithm with selective training data updates is implemented to establish and refine the FBG-empowered (F-emp) pose estimator online . The integration of F-emp pose estimation can improve sensing robustness by reducing the number of times that visual tracking is lost given moving visual obstacles and varying lighting. In particular, this integration solves pose estimation failures under full occlusion of the tracked features or complete darkness. Utilizing the fused pose feedback, a hybrid controller incorporating kinematics and data-driven algorithms is proposed to accomplish fast convergence with high accuracy. The online-learning error compensator can improve the target tracking performance with a 52.3%–90.1% error reduction compared with constant-curvature model-based control, without requiring fine model-parameter tuning and prior data acquisition.

This article is focused on the design of an output tracking control scheme for a class of continuous-time periodic piecewise time-varying systems (PPTVSs) with actuator saturation and nonlinear perturbations. The energy-to-peak tracking performance is studied based on an equivalent condition on the definiteness property of matrix polynomials. Considering the actuator saturation and nonlinear perturbation, matrix polynomial-based sufficient conditions are derived through the Lyapunov method using periodic matrix functions. From a perspective of subinterval segmentation aimed at PPTVSs, the proposed conditions can achieve less conservatism for tracking the output of a periodic time-varying reference system, while the controller gains can be computed using convex optimization. Moreover, a heuristic algorithm is constructed to simultaneously guarantee the closed-loop state convergence and the output tracking performance. The reduction in conservatism and the effectiveness of algorithm are demonstrated by illustrative case studies.

Although the soft robot is gaining considerable popularity in dexterous and safe manipulation, accurate motion control is still an open problem to be explored. Recent investigations suggest that reinforcement learning (RL) is a promising solution but lacks efficient adaptability for Sim2Real transfer or environment variations. In this paper, we present a deep deterministic policy gradient (DDPG)-based control system for the continuous task-space manipulation of soft robots. Domain randomization is adopted in simulation for fast control-policy initialization, while an offline retraining strategy is utilized to update the controller parameters for incremental learning. The experiments demonstrate that the proposed RL controller can track a moving target accurately (with RMSE of 1.26 mm), and accommodate to external varying load effectively (with ~30% RMSE reduction after retraining). Comparisons among the proposed RL controller and other supervised-learning-based controllers in handling additional tip load were also conducted. The results support that our RL method is appropriate for automatic learning such that there is no need of manual interference for data processing, particularly in cases with external disturbances and actuation redundancy.