Haoyu Liu

Haoyu Liu

Auto-generated profile, based on publicly available information

Haoyu Liu

Research Center for Intelligent Robotics, Zhejiang Lab, Shanghai, China

Research Focus & Metrics

Academic Career Length

7Years (2020-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
16 May 2024 in IEEE Transactions on Knowledge and Data Engineering
Verified
Journal Article
Closed access
YSYaoxian Song
Yaoxian Song
PSPenglei Sun
Penglei Sun
HLHaoyu Liu
Haoyu Liu
ZLZhixu Li
Zhixu Li
WSWei SONG
Wei SONG
XZXiaofang Zhou
Xiaofang Zhou
Embodied AI is one of the most popular studies in artificial intelligence and robotics, which can effectively improve the intelligence of real-world agents (i.e. robots) serving human beings. Scene knowledge is important for an agent to understand the surroundings and make correct decisions in the varied open world. Currently, knowledge base for embodied tasks is missing and most existing work use general knowledge base or pre-trained models to enhance the intelligence of an agent. For conventional knowledge base, it is sparse, insufficient in capacity and cost in data collection. For pre-trained models, they face the uncertainty of knowledge and hard maintenance. To overcome the challenges of scene knowledge, we propose a scene-driven multimodal knowledge graph (Scene-MMKG) construction method combining conventional knowledge engineering and large language models. A unified scene knowledge injection framework is introduced for knowledge representation. To evaluate the advantages of our proposed method, we instantiate Scene-MMKG considering typical indoor robotic functionalities ( Manip ulation and Mob ility), named ManipMob-MMKG . Comparisons in characteristics indicate our instantiated ManipMob-MMKG has broad superiority on data-collection efficiency and knowledge quality. Experimental results on typical embodied tasks show that knowledge-enhanced methods using our instantiated ManipMob-MMKG can improve the performance obviously without re-designing model structures complexly.

3 January 2020 in Water, Air, & Soil Pollution
Verified
Journal Article
Closed access
PDF file
FWFei Wang
Fei Wang
WXWangqi Xu
Wangqi Xu
ZXZhaochen Xu
Zhaochen Xu
HLHaoyu Liu
Haoyu Liu

21 October 2024
Verified
Conference Paper
Open access
KZKaichen Zhao
Kaichen Zhao
YSYaoxian Song
Yaoxian Song
HZHaiquan Zhao
Haiquan Zhao
HLHaoyu Liu
Haoyu Liu
TLTiefeng Li
Tiefeng Li
ZLZhixu Li
Zhixu Li
Visual language navigation (VLN) is one of the important research in embodied AI. It aims to enable an agent to understand the surrounding environment and complete navigation tasks. VLN instructions could be categorized into coarse-grained and fine-grained commands. Fine-grained command describes a whole task with subtasks step-by-step. In contrast, coarse-grained command gives an abstract task description, which more suites human habits. Most existing work focuses on the former kind of instruction in VLN tasks, ignoring the latter abstract instructions belonging to daily life scenarios. To overcome the above challenge in abstract instruction, we attempt to consider coarse-grained instruction in VLN by event knowledge enhancement. Specifically, we first propose a prompt-based framework to extract an event knowledge graph (named VLN-EventKG =) for VLN integrally over multiple mainstream benchmark datasets. Through small and large language model collaboration, we realize knowledge-enhanced navigation planning (named EventNav) for VLN tasks with coarse-grained instruction input. Additionally, we design a novel dynamic history backtracking module to correct potential error action planning in real time. Experimental results in various public benchmarks show our knowledge-enhanced method has superiority in coarse-grained-instruction VLN using our proposed VLN-EventKG with over 5% improvement in success rate. Our project is available at https://sites.google.com/view/vln-eventkg