Liangsheng Lu

Liangsheng Lu

Auto-generated profile, based on publicly available information

Liangsheng Lu

University of Science and Technology of China, Anhui, China

Research Focus & Metrics

Academic Career Length

5Years (2022-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
17 August 2022 in IEEE Transactions on Artificial Intelligence
Verified
Journal Article
Closed access
LLLiangsheng Lu
Liangsheng Lu
WZWei Zhai
Wei Zhai
HLHongchen Luo
Hongchen Luo
YKYu Kang
Yu Kang
YCYang Cao
Yang Cao
Affordance detection, which refers to perceiving objects with potential action possibilities in images, is a challenging task since the possible affordance depends on the person's purpose in real-world application scenarios. The existing works mainly extract the inherent human-object dependencies from image/video to accommodate affordance properties that change dynamically. In this paper, we explore to perceive affordances from a vision-language perspective, and consider the challenging phrase-based affordance detection task, i.e., given a set of phrases describing the potential actions, all the object regions in a scene with the same affordance should be detected. To this end, we propose a Cyclic Bilateral Consistency Enhancement Network (CBCE-Net) to align language and vision features in a progressive manner. Specifically, the presented CBCE-Net consists of a mutual guided vision-language module that updates the common features of vision and language in a progressive manner, and a Cyclic Interaction Module (CIM) that facilitates the perception of possible interaction with objects in a cyclic manner. In addition, we extend the public Purpose-driven Affordance Dataset (PAD) by annotating affordance categories with short phrases. The extensive contrastive experimental results demonstrate the superior performance of our method over nine typical methods from four relevant fields in terms of both objective metrics and visual quality. The related code and dataset will be released at https://github.com/lulsheng/CBCE-Net. Impact Statement—Affordance learning is concerned with the possible set of actions that an environment can offer to an actor. At present, almost all work focus on utilizing cues of human-object interactions in visual media such as images and videos to learn affordances. Few work explores affordance using language information, which is crucial to building more intelligent agents because we live a multimodal world. In this paper, we propose a phrase-based affordance detection task combining vision and language modalities. We first build a vision-language dataset based on the public Purpose-driven Affordance Dataset (PAD). Then we propose a method to fuse and align multi-modal features. Our method outperforms relevant methods from several other fields in terms of both objective metrics and visual quality. The proposed dataset and method can facilitate building more intelligent agents that can better comprehend the surrounding environment. For instance, the phase-based affordance detection can be used in building more intelligent home service systems. The agents can understand instructions of instructions without specific entities.