Changchong Sheng

Changchong Sheng

Auto-generated profile, based on publicly available information

Changchong Sheng

National University of Defense Technology (NUDT)

Research Focus & Metrics

Academic Career Length

6Years (2021-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
16 August 2021 in IEEE Transactions on Multimedia
Verified
Journal Article
Open access
CSChangchong Sheng
Changchong Sheng
XZXinzhong Zhu
Xinzhong Zhu
HXHuiying Xu
Huiying Xu
MPMatti Pietikainen
Matti Pietikainen
LLLi Liu
Li Liu
The goal of this work is to recognize words, phrases, and sentences being spoken by a talking face without given the audio. Current deep learning approaches for lip reading focus on exploring the appearance and optical flow information of videos. However, these methods do not fully exploit the characteristics of lip motion. In addition to appearance and optical flow, the mouth contour deformation usually conveys significant information that is complementary to others. However, the modeling of dynamic mouth contour has received little attention than that of appearance and optical flow. In this work, we propose a novel model of dynamic mouth contours called Adaptive Semantic-Spatio-Temporal Graph Convolution Network (ASST-GCN), to go beyond previous methods by automatically learning both the spatial and temporal information from videos. To combine the complementary information from appearance and mouth contour, a two-stream visual front-end network is proposed. Experimental results demonstrate that the proposed method significantly outperforms the state-of-the-art lip reading methods on several large-scale lip reading benchmarks.

12 July 2023 in ACM Transactions on Multimedia Computing, Communications, and Applications
Verified
Journal Article
Open access
RCRuoyu Chen
Ruoyu Chen
JLJingzhi Li
Jingzhi Li
HZHua Zhang
Hua Zhang
CSChangchong Sheng
Changchong Sheng
LLLi Liu
Li Liu
XCXiaochun Cao
Xiaochun Cao
Recently, we have witnessed substantial success using the deep neural network in many tasks. While there still exists concerns about the explainability of decision-making, it is beneficial for users to discern the defects in the deployed deep models. Existing explainable models either provide the image-level visualization of attention weights or generate textual descriptions as post-hoc justifications. Different from existing models, in this paper, we propose a new interpretation method that explains the image similarity models by salience maps and attribute words. Our interpretation model contains visual salience maps generation and the counterfactual explanation generation. The former has two branches: global identity relevant region discovery and multi-attribute semantic region discovery. Branch one aims to capture the visual evidence supporting the similarity score, which is achieved by computing counterfactual feature maps. Branch two aims to discover semantic regions supporting different attributes, which helps to understand which attributes in an image might change the similarity score. Then, by fusing visual evidence from two branches, we can obtain the salience maps indicating important response evidence. The latter will generate the attribute words that best explain the similarity using the proposed erasing model. The effectiveness of our model is evaluated on the classical face verification task. Experiments conducted on two benchmarks VGG-Face2 and Celeb-A demonstrate that our model can provide convincing interpretable explanations for the similarity. Moreover, our algorithm can be applied to evidential learning cases, e.g. finding the most characteristic attributes in a set of face images and we verify its effectiveness on the VGGFace2 dataset.