Embodied AI is one of the most popular studies in artificial intelligence and robotics, which can effectively improve the intelligence of real-world agents (i.e. robots) serving human beings. Scene knowledge is important for an agent to understand the surroundings and make correct decisions in the varied open world. Currently, knowledge base for embodied tasks is missing and most existing work use general knowledge base or pre-trained models to enhance the intelligence of an agent. For conventional knowledge base, it is sparse, insufficient in capacity and cost in data collection. For pre-trained models, they face the uncertainty of knowledge and hard maintenance. To overcome the challenges of scene knowledge, we propose a scene-driven multimodal knowledge graph (Scene-MMKG) construction method combining conventional knowledge engineering and large language models. A unified scene knowledge injection framework is introduced for knowledge representation. To evaluate the advantages of our proposed method, we instantiate Scene-MMKG considering typical indoor robotic functionalities ( Manip ulation and Mob ility), named ManipMob-MMKG . Comparisons in characteristics indicate our instantiated ManipMob-MMKG has broad superiority on data-collection efficiency and knowledge quality. Experimental results on typical embodied tasks show that knowledge-enhanced methods using our instantiated ManipMob-MMKG can improve the performance obviously without re-designing model structures complexly.

Good transparency performance (including position tracking and force feedback) is a significant issue for teleoperation control design. The four-channel control approach, which is based on the impedance matching between the master and slave, is an effective method to achieve the perfect transparency performance of teleoperation system. However, the obvious drawback of this design is that the time delay existing in the communication channel is neglected and thus the stability of this design under time delay cannot be guaranteed. Some passivity based control approaches such as wave-variable are developed to handle the stability problem of teleoperation system under time delay, and have been tried to add into four-channel control design. Though the stability under time delay is achieved for this kind of design, the transparency performance may largely decrease since the traditional wave-variable may essentially suffer some performance limitations (e.g., wave reflection). Thus, simultaneously realizing the stability and good transparency performance for four channel teleoperation system is still a challenging issue. In this paper, an improved wave-variable based four-channel control design is developed for bilateral teleoperation system to guarantee stability and further improve transparency performance. Under four channel control architecture, the passivity based time-delay compensator by the modified wave transform and the local force feedback are provided to compensate the distortion by reducing wave reflection and further enhance the stability and transparency of teleoperation system. The experiment is carried out, and the results show the great improvement of the proposed method to achieve the good transparency performance compared to the previous four-channel control designs.

Underwater manipulators are important robotic tools in the exploration of the ocean environment. Up to now, most existing underwater manipulators are rigid and with fixed 5 or 7 degrees of freedom (DOF), which may not be very suitable for some complicated underwater scenarios (e.g., pipe networks, narrow deep cavities, etc.). The biomimetic concept of muscles and tendons is also considered as continuum manipulators, but load capacity and operation accuracy are their essential drawbacks and thus limit their practical applications. Recently, the cable-driven technique has been developed for manipulators, which can include numerous joints and hyper-redundant DOF to execute tasks with dexterity and adaptability and thus they have strong potential for these complex underwater applications. In this paper, the design of a novel cable-driven hyper-redundant manipulator (CDHRM) is introduced, which is driven by multiple cables passing through the tubular structure from the base to the end-effector, and the joint numbers can be extended and decided by the specific underwater task requirements. The kinematic analysis of the proposed CDHRM is given which includes two parts: the cable-joint kinematics and the joint-end kinematics. The geometric relationship between the cable length and the joint angles are derived via the established geometric model for the cable-joint kinematics, and the projection relationship between the joint angles and end-effector’s pose is established via the spatial coordinate transformation matrix for the joint-end kinematics. Thus, the complex mapping relationships among the cables, joints and end-effectors are clearly achieved. To implement precise control, the kinematic control scheme is developed for the CDHRM with series-parallel connections and hyper-redundancy to achieve good tracking performance. The experiment on a real CDHRM system with five joints is carried out and the results verify the accuracy of kinematics solution, and the effectiveness of the proposed control design. Particularly, three experiments are tested in the underwater environment, which verifies its good tracking performance, load carrying and grasping capacity.

In this work, we present a visual-inertial-pressure (VIP) fusion method for underwater robot localization. Specifically, this letter focuses on the tightly-coupled fusion of pressure measurements into a visual inertial odometry (VIO) based on sliding window optimization. Previous works used to associate partial pressure measurements with the nearest keyframes, which not only fail to utilize all the pressure measurement information but also introduce measurement errors that only lead to sub-optimal solutions. Inspired by the current tightly-coupled visual-inertial-GPS and visual-inertial-UWB fusion methods, this letter uses IMU preintegration algorithm to derive the pressure factors. Furthermore, we propose a backward IMU preintegration method and the pressure factors are derived using forward or backward IMU preintegration based on the time-offset between the pressure measurements and the adjacent keyframes. Quantitative and qualitative analyses through simulation and real-world datasets experiments demonstrate the effectiveness of the method with negligible time cost.

Bilateral teleoperation technique has drew much attention with its attractive superiority to implement the tasks in hazardous environments. Transmission delays and uncertainties are the two main challenges in the nonlinear bilateral teleoperation system to guarantee stability and achieve good transparency performance (including position tracking and force feedback) simultaneously. In this paper, an RBFNN-based adaptive sliding mode control design is developed for nonlinear bilateral teleoperation system with transmission delays and uncertainties. For details, the reference trajectory producer is designed in both the master and slave sides, and produce the passive reference trajectories for the tracking of master/slave manipulators. The RBFNN-based adaptive sliding mode controller is designed separately for the master and slave to achieve the good tracking performance under system uncertainties. To mitigate the negative effect of transmission delays on the system’s stability, a projection mapping by saturation function is applied in the master side to guarantee the boundedness of the delayed environmental torque. Thus, the global stability and good transparency performance with both position tracking and force feedback can be simultaneously achieved for our proposed method. The comparative experiment is carried out and the results show the significant performance improvement with our proposed control design.