In this paper, a robotic system based on intraoperative 2D fluoroscopy navigation in which fluoroscopy images are obtained by C-arm popularized in the clinic, for spine surgery is proposed. The procedure of robot-assisted spine surgery is designed and described. An improved special 6-dof hybrid robot considering both the stiffness and workspace is proposed. And the kinematic model and jacobian matrix of robot are established. To realize fluroscopy images acquisition and avoid interfering with the patient during the process of positioning, human-robot cooperation based on fuzzy variable admittance control considering arm stiffness which would influence the stability of robot, is applied in robotic system. In addition, navigation techniques containing a novel bi-planar design, a robust circular control point detector and closed-loop positioning of tool, based on fluoroscopy images are proposed. In experiments, kinematic model of hybrid robot is test based on laser tracker and the distance error is 0.17 mm which is much smaller than the motion range. The effect of tool weight compensation is measured and the max mean force and torque errors are 0.7477N and 0.0617 N.M. The experimental results could meet requirements in human-robot cooperation. Furthermore, we test the the robotic system proposed in this paper on spine model bone and cadaver with the help of surgeons. The experimental results are validated by 2D image c-arm-based and 3D image CT-based and considered clinically acceptable.

Vision-and-Language Navigation (VLN) has garnered widespread attention and research interest due to its potential applications in real-world scenarios. Despite significant progress in the VLN field in recent years, limitations persist. Many agents struggle to make accurate decisions when faced with similar candidate views during navigation, relying solely on the overall features of these views. This challenge primarily arises from the lack of common-sense knowledge about room layouts. Recognizing that room knowledge can establish relationships between rooms and objects in the environment, we construct room layout knowledge described in natural language by leveraging BLIP-2, including relationships between rooms and individual objects, relationships between objects, attributes of individual objects (such as color), and room types, thus providing comprehensive room layout information to the agent. We propose a Knowledge-Enhanced Scene Understanding (KESU) model to augment the agent's understanding of the environment by leveraging room layout knowledge. The Instruction Augmentation Module (IA) and the Knowledge History Fusion Module (KHF) in KESU respectively provide room layout knowledge for instructions and vision-history features, thereby enhancing the agent's navigation abilities. To more effectively integrate knowledge information with instruction features, we introduce Dynamic Residual Fusion (DRF) in the IA module. Finally, we conduct extensive experiments on the R2R, REVERIE, and SOON datasets, demonstrating the effectiveness of the proposed approach

In robot-assisted live-line maintenance, bilateral teleoperation is still a popular and effective approach in assisting operators to accomplish hazards tasks. Particularly, teleoperation under overhead power lines attach greater expectation on safe operation and telepresence. In this article, we propose a visual-haptic bilateral teleoperation strategy, i.e., EMGP-VH , based on visual guidance, haptic constraint and mixed reality (MR) augmentation. To the best of our knowledge, electromagnetic field is first applied to serve the path planning of teleoperation in live-line maintenance. In visual guidance, EMG-potential fields are integrated into RRT* to calculate a low-energy path. At the same time, real-time haptic constraint is calculated based on a tube virtual fixture. MR augmentation also works as an indispensable part in both the platform construction and visual guidance. Our proposal has been extensively compared using seven objective performances and three subjective questionnaires both in simulation and real-world experiment with five different scenes and two approaches state-of-the-art, respectively. Functionality of EMGP-RRT* and effectiveness of haptic constraint are further analyzed. Results show that EMGP-RRT* has significant improvements both in searching efficiency and safety performances; and the proposed system ( EMGP-VH ) significantly contributes to improving telepresence and ensuring safe operations during live-line maintenance, resulting in a 30% reduction in operation time and a 60% decrease in trajectory offset.

To improve the robotic assembly effects in unstructured environments, a reinforcement learning (RL) algorithm is introduced to realize a variable admittance control. In this article, the mechanisms of a peg-in-hole assembly task and admittance model are first analyzed to guide the control strategy and experimental parameters design. Then, the admittance parameter identification process is defined as the Markov decision process (MDP) problem and solved with the RL algorithm. Furthermore, a fuzzy reward system is established to evaluate the action–state value to solve the complex reward establishment problem, where the fuzzy reward includes a process reward and a failure punishment. Finally, four sets of experiments are carried out, including assembly experiments based on the position control, fuzzy control, and RL algorithm. The necessity of compliance control is demonstrated in the first experiment. The advantages of the proposed algorithms are validated by comparing them with different experimental results. Moreover, the generalization ability of the RL algorithm is tested in the last two experiments. The results indicate that the proposed RL algorithm effectively improves the robotic compliance assembly ability.

Adversarial Inverse Reinforcement Learning (AIRL) has gained popularity as an alternative to supervised imitation learning, addressing the distributional bias issue of the latter. However, it still faces significant challenges in long-horizon tasks due to the lack of effective exploration. In our letter, we demonstrate that standard AIRL strategies end exploration prematurely during online reinforcement learning and fail to learn the entire task due to their inability to fully conform to the expert distribution, which is particularly detrimental to real-world robots. To address these challenges, we introduce the SC-AIRL approach. It decomposes long-horizon tasks into logical subtasks which reduces the agent's need for rich exploration. SC-AIRL utilizes expert demonstrations for performing multiple subtasks and shares a single critic and identical reward function across different subtask trainings. Additionally, we incorporate a human intervention mechanism during the subtask learning process to keep exploration from ending prematurely. Our experiments in challenging robot manipulation tasks demonstrate that SC-AIRL outperforms our baselines significantly. Furthermore, we conduct an exploratory experiment and an empirical analysis, emphasizing the potential of the model to manage complex tasks and the advantages of SC-AIRL over the baseline, respectively.

6D pose estimation has been pervasively applied to various robotic applications, such as service robots, collaborative robots, and unmanned warehouses. However, accurate 6D pose estimation is still a challenge problem due to the complexity of application scenarios caused by illumination changes, occlusion and even truncation between objects, and additional refinement is required for accurate 6D object pose estimation in prior work. Aiming at the efficiency and accuracy of 6D object pose estimation in these complex scenes, this paper presents a novel end‐to‐end network, which effectively utilises the contextual information within a neighbourhood region of each pixel to estimate the 6D object pose from RGB‐D images. Specifically, our network first applies the attention mechanism to extract effective pixel‐wise dense multimodal features, which are then expanded to multi‐scale dense features by integrating pixel‐wise features at different scales for pose estimation. The proposed method is evaluated extensively on the LineMOD and YCB‐Video datasets, and the experimental results show that the proposed method is superior to several state‐of‐the‐art baselines in terms of average point distance and average closest point distance.