The goal of this work is to recognize words, phrases, and sentences being spoken by a talking face without given the audio. Current deep learning approaches for lip reading focus on exploring the appearance and optical flow information of videos. However, these methods do not fully exploit the characteristics of lip motion. In addition to appearance and optical flow, the mouth contour deformation usually conveys significant information that is complementary to others. However, the modeling of dynamic mouth contour has received little attention than that of appearance and optical flow. In this work, we propose a novel model of dynamic mouth contours called Adaptive Semantic-Spatio-Temporal Graph Convolution Network (ASST-GCN), to go beyond previous methods by automatically learning both the spatial and temporal information from videos. To combine the complementary information from appearance and mouth contour, a two-stream visual front-end network is proposed. Experimental results demonstrate that the proposed method significantly outperforms the state-of-the-art lip reading methods on several large-scale lip reading benchmarks.

Dynamic ensemble has significantly greater potential space to improve the classification of imbalanced data compared to static ensemble. However, dynamic ensemble schemes are far less successful than static ensemble methods in the imbalanced learning field. Through an in-depth analysis on the behavior characteristics of dynamic ensemble, we find that there are some important problems that need to be addressed to release the full potential of dynamic ensemble, including but not limited to, correcting the component classifiers' bias towards the majority classes, increasing the proportions of the positive classifiers (i.e., the component classifiers making correct prediction) for difficult samples, and providing the accurate competence estimations on the hard-to-classify samples w.r.t the classifier pool. Inspired by these, we propose a Dynamic Ensemble Framework for imbalanced data classification (imDEF). imDEF first uses the data generation method OREMG to generate multiple artificial synthetic datasets, which have diverse class distributions by rebalancing the original imbalanced data. Based on each of such synthetic datasets, imDEF then utilizes a Classification Error-aware Self-Paced Sampling Ensemble (SPSECE) method to gradually focus more on difficult samples, to create a low-biased classifier pool and increase the proportions of the positive classifiers for the difficult samples. Finally, imDEF constructs a referee system to achieve the competence estimations by leveraging an Ensemble Margin-aware Self-Paced Sampling Ensemble (SPSEEM) method. SPSEEM incrementally strengthens the learning of the hard-to-classify samples, so that the competent levels of component classifiers could be estimated accurately. Extensive experiments demonstrate the effectiveness of imDEF. The source codes have been made publicly available on GitHub

Multiview clustering (MVC) collects complementary and abundant information, which draws much attention in machine learning and data mining community. Existing MVC methods usually hold the assumption that all the views are complete. However, multiple source data are often incomplete in real‐world applications, and so on sensor failure or unfinished collection process, which gives rise to incomplete multiview clustering (IMVC). Although enormous efforts have been devoted in IMVC, there still are some urgent issues that need to be solved: (i) The locality among multiple views has not been utilized in the existing mechanism; (ii) Existing methods inappropriately force all the views to share consensus representation while ignoring specific structures. In this paper, we propose a novel method termed partial MVC with locality graph regularization to address these issues. First, followed the traditional IMVC approaches, we construct weighted semi‐nonnegative matrix factorization models to handle incomplete multiview data. Then, upon the consensus representation matrix, the locality graph is constructed for regularizing the shared feature matrix. Moreover, we add the coefficient regression term to constraint the various base matrices among views. We incorporate the three aforementioned processes into a unified framework, whereas they can negotiate with each other serving for learning tasks. An effective iterative algorithm is proposed to solve the resultant optimization problem with theoretically guaranteed convergence. The comprehensive experiment results on several benchmarks demonstrate the effectiveness of the proposed method.