Intraoperative soft tissue manipulation is a critical challenge in autonomous robotic surgery. Furthermore, the intricate in vivo environment surrounding the target soft tissues poses additional hindrances to autonomous robotic decision-making. Previous studies assumed the grasping point was known and the target deformation could be achieved. The constraints were assumed to be constant during the operation, and there were no obstacles around the soft tissue. To address these problems, an intuition-guided deep reinforcement learning framework based on soft actor-critic (ID-SAC) was proposed for soft tissue manipulation under unknown constraints. The SAC algorithm is automatically activated upon encountering an obstacle, and the designed intuitive manipulation (IM) strategy is used to pull soft tissues toward the target shape directly when the obstacle is distant. A regulator factor is designed as an action within this framework to coordinate the IM approach and the SAC network. A reward function is designed to balance the exploration and exploitation of large deformations. Simultaneously, we proposed an autonomous grasp point selection neural network to prevent the impractical selection of grasp points, ensuring they can reach the target while avoiding grasping lesions and constrained areas. Successful simulation results confirmed that the proposed framework can manipulate the soft tissue while avoiding obstacles and adding new positional constraints. Compared with the SAC algorithm, the proposed framework can markedly increase the robotic soft tissue manipulation ability by automatically adjusting the regulator factors.
23 November 2023 in IEEE Journal of Biomedical and Health Informatics
Untreated pain in critically ill patients can lead to immunosuppression and increased metabolic activity, with severe clinical consequences such as tachypnea and delirium. Continuous pain assessment is challenging due to nursing shortages and intensive care unit (ICU) workload. Mechanical ventilation equipment obscures the facial features of many patients in the ICU, making previous facial pain detection methods based on full-face images inapplicable. This paper proposes a facial Action Units (AUs) guided pain assessment network for faces under occlusion. The network consists of an AU-guided (AUG) module, a texture feature extraction (TFE) module, and a pain assessment (PA) module. The AUG module automatically detects AUs in the non-occluded areas of the face. In contrast, the TFE module detects the facial landmarks and crops prior knowledge patches, a random exploration patch, and a global feature patch. Then these patches are fed into two convolutional networks to extract texture features. Afterward, the designed AU guidances and texture features are fused in the PA module to assess the pain state. Extensive validation is conducted on a public dataset and two datasets created in this work. The proposed network architecture achieves superior performance in binary classification, four-class classification, and intensity regression tasks. In addition, we have successfully applied the network to actual data collected in the laboratory environment with excellent results.
Tire tread pattern image classification plays an important role in crime scene and traffic accident investigation. Due to the lack of standard test dataset, there is little work done in this area. For efficient texture feature description, inherent characteristic of tire patterns need to be considered. Leveraging on the directionality characteristics of tread patterns, a novel texture feature extraction algorithm is proposed based on adaptive weighted feature fusion with the weights defined by sub-band energy ratio. The proposed approach consists of: (1) discrete wavelet decomposition of tire tread image to obtain low frequency, horizontal, vertical and diagonal sub-bands; (2) extraction of rotation-invariant uniform local binary pattern features from the sub-band images; (3) concatenating the tread pattern directional features, weighted by their corresponding sub-band energies. Applying SVM for tire tread pattern classification, experimental results on real-world tire tread patterns show that the proposed texture feature extraction algorithm is outperforms other prior methods.
14 March 2025 in IEEE Transactions on Instrumentation and Measurement
Robotic ultrasound systems can potentially release the workload of sonographers and overcome operator-dependent limitations. Previous methods of ultrasound image localization without mapping, constrained by position and specific image features, have limited applicability. This paper presents Ultra-Malin, a mapping and localization method for autonomous robotic ultrasound using implicit neural representations. Our method leverages its inter-image inference capacity to perform image-based localization without specifying tissue features. Similar to a pinhole camera model, a geometric model is first proposed to determine scanning depth, as ultrasound transducer rays can be analogous to rays from a camera. Subsequently, we retrieve the 5-dimensional (5D) coordinates of ultrasonic ray points and employ volume rendering techniques to render the pixel color of the ultrasound image. A global search method with density-guided ray sampling is then presented to locate the ultrasound image by minimizing the error between the target and the rendered images. The experimental results show that the rendered images for probe tilting, rotational, and linear scanning have structural similarity (SSIM) of 0.7788, 0.7999, and 0.8421, respectively, outperforming existing methods using only a small amount of data. The localization experiments verify that the probe can accurately locate the input image and the rendered ultrasound images are highly consistent with the actual images. Furthermore, the probe can still be located at the target position even if disturbances obscure part features, and the search space can be enlarged by the proposed density-guided ray sampling.
24 October 2024 in IEEE Robotics and Automation Letters
Accurately modeling the interactions between deformable linear objects (DLOs) and their environments is crucial for active deformation control by robot manipulators. Graph Neural Networks (GNNs) have shown immense potential in the particle-based dynamics of DLOs. However, most existing studies propagate particle information in sequence, ignoring that particle motions, including the distal particle, correlate strongly with each other and the interaction state. In this paper, a global and local attention dynamic model named GladNet is designed based on GNNs and the attention mechanism to aggregate information among particles and focus on the interaction particles for DLO interaction with the environment. Specifically, a global virtual node is proposed to deliver particle information and shorten the propagation path for the first time, which connects all the particles and aggregates global information. When the DLOs and the obstacle boundary particles are close, an edge is established between them to capture the interaction state. Moreover, we group all the particles by $k$ -hop neighbors and design a HopSA module that combines hop attention and self-attention to discover the correlates among adjacent particles. Experimental results on simulation and real-world data show that the proposed GladNet network's predictive accuracy outperforms baseline models, especially in long-term prediction.