Lipreading aims to recognize sentences being spoken by a talking face. In recent years, the lipreading method has achieved a high level of accuracy on large datasets and made breakthrough progress. However, lipreading is still far from being solved, and existing methods tend to have high error rates on the wild data and have the defects of disappearing training gradient and slow convergence. To overcome these problems, we proposed an efficient end-to-end sentence-level lipreading model, using an encoder based on a 3D convolutional network, ResNet50, Temporal Convolutional Network (TCN), and a CTC objective function as the decoder. More importantly, the proposed architecture incorporates TCN as a feature learner to decode feature. It can partly eliminate the defects of RNN (LSTM, GRU) gradient disappearance and insufficient performance, and this yields notable performance improvement as well as faster convergence. Experiments show that the training and convergence speed are 50% faster than the state-of-the-art method, and improved accuracy by 2.4% on the GRID dataset.

Lipreading is a visual recognition of speech by using lip movement, which aims to recognise phrases and sentences spoken by a talking face without the audio. However, the existed models for lipreading suffer from slow training speed and insufficient performance. To accelerate the training speed of the model for lipreading, a batch group training algorithm is proposed, which groups all the data of different frames. In addition, a 3D‐MouthNet‐BLSTM‐CTC architecture for lipreading is proposed to improve model performance. It bases on a 3D convolutional neural network, MouthNet, two Bi‐LSTMs, and a CTC objective function. Experiment results in Oulu‐VS2 and self‐built dataset show that 96.2% accuracy rate is achieved on the Oulu‐VS2 dataset, and 93.8% accuracy rate is achieved on the GRID dataset. This article is about lipreading research. It mainly uses deep learning methods to study lip‐reading. A new network architecture and tests on public data sets are proposed to achieve the best results.