23 November 2022 in IEEE Transactions on Knowledge and Data Engineering
Recent years have witnessed the resurgence of knowledge engineering which is featured by the fast growth of knowledge graphs. However, most of existing knowledge graphs are represented with pure symbols, which hurts the machine's capability to understand the real world. The multi-modalization of knowledge graphs is an inevitable key step towards the realization of human-level machine intelligence. The results of this endeavor are Multi-modal Knowledge Graphs (MMKGs). In this survey on MMKGs constructed by texts and images, we first give definitions of MMKGs, followed with the preliminaries on multi-modal tasks and techniques. We then systematically review the challenges, progresses and opportunities on the construction and application of MMKGs respectively, with detailed analyses of the strengths and weaknesses of different solutions. We finalize this survey with open research problems relevant to MMKGs.
16 May 2024 in IEEE Transactions on Knowledge and Data Engineering
Embodied AI is one of the most popular studies in artificial intelligence and robotics, which can effectively improve the intelligence of real-world agents (i.e. robots) serving human beings. Scene knowledge is important for an agent to understand the surroundings and make correct decisions in the varied open world. Currently, knowledge base for embodied tasks is missing and most existing work use general knowledge base or pre-trained models to enhance the intelligence of an agent. For conventional knowledge base, it is sparse, insufficient in capacity and cost in data collection. For pre-trained models, they face the uncertainty of knowledge and hard maintenance. To overcome the challenges of scene knowledge, we propose a scene-driven multimodal knowledge graph (Scene-MMKG) construction method combining conventional knowledge engineering and large language models. A unified scene knowledge injection framework is introduced for knowledge representation. To evaluate the advantages of our proposed method, we instantiate Scene-MMKG considering typical indoor robotic functionalities ( Manip ulation and Mob ility), named ManipMob-MMKG . Comparisons in characteristics indicate our instantiated ManipMob-MMKG has broad superiority on data-collection efficiency and knowledge quality. Experimental results on typical embodied tasks show that knowledge-enhanced methods using our instantiated ManipMob-MMKG can improve the performance obviously without re-designing model structures complexly.
2 May 2025 in IEEE Transactions on Automation Science and Engineering
Robotic grasping is a fundamental ability for a robot to interact with the environment. Current methods focus on how to obtain a stable and reliable grasping pose in object level, while little work has been studied on part (shape)-wise grasping which is related to fine-grained grasping and robotic affordance. Parts can be seen as atomic elements to compose an object, which contains rich semantic knowledge and a strong correlation with affordance. However, lacking a large part-wise 3D robotic dataset limits the development of part representation learning and downstream applications. In this paper, we propose a new large Language-guided SHape grAsPing datasEt (named LangSHAPE) to promote 3D part-level affordance and grasping ability learning. From the perspective of robotic cognition, we design a two-stage fine-grained robotic grasping framework (named LangPartGPD), including a novel 3D part language grounding model and a partaware grasp pose detection model, in which explicit language input from human or large language models (LLMs) could guide a robot to generate part-level 6-DoF grasping pose with textual explanation. Our method combines the advantages of humanrobot collaboration and LLMs’ planning ability using explicit language as a symbolic intermediate. To evaluate the effectiveness of our proposed method, we perform 3D part grounding and fine-grained grasp detection experiments on both simulation and physical robot settings, following language instructions across different degrees of textual complexity. Results show our method achieves competitive performance in 3D geometry fine-grained grounding, object affordance inference, and 3D part-aware grasping tasks. Our dataset and code are available on our project website https://sites.google.com/view/lang-shape.
31 July 2023 in E3S Web of Conferences
The permanent ground displacement of geological disasters is the main reason inducing large additional axial stress of pipelines. This paper presents a surface displacement measurement method above the pipeline based on SBAS-InSAR remote sensing technology, which realizes millimeter-level high-precision monitoring of surface displacement of pipeline in kilometer range. Based on the spatial coordinate data of the pipeline center line and the geometric feature data of the pipeline, combined with the cubic spline interpolation algorithm, a reconstruction method of the pipeline three-dimensional space routing is proposed. Considering the real routing of the pipeline and the process load conditions such as temperature and pressure, the finite element model of pipeline stress analysis is established by using the space pipe element and the nonlinear pipe soil element. Taking the remote sensing landslide displacement as the accurate boundary condition, the calculation method of pipeline stress in landslide section is proposed. The practice of a high-risk landslide section in Liangxiang-Xishatun of Shaanxi-Beijing Pipeline from 2021 to 2022 shows that the maximum relative error between the pipeline stress results obtained by this method and the distributed strain gauge monitoring results does not exceed 16%, which can effectively realize the spatial reconstruction and safety evaluation of the pipeline stress state in the landslide section, and provide a technical basis for the establishment of digital twins of pipeline safety state in future geological disaster sections.