Real-time driving behavior identification has a wide range of applications in monitoring driver states and predicting driving risks. In contrast to the traditional approaches that were mostly based on a single data source with poor identification capabilities, this paper innovatively integrates driver expression into driving behavior identification. First, 12-day online car-hailing driving data were collected in a non-intrusive manner. Then, with vehicle kinematic data and driver expression data as inputs, a stacked Long Short-Term Memory (S-LSTM) network was constructed to identify five kinds of driving behaviors, namely, lane keeping, acceleration, deceleration, turning, and lane changing. The Artificial Neural Network (ANN) and XGBoost algorithms were also employed as a comparison. Additionally, ten sliding time windows of different lengths were introduced to generate driving behavior identification samples. The results show that, using all sources of data yields better results than using the kinematic data only, with the average F1 value improved by 0.041, while the S-LSTM algorithm is better than the ANN and XGBoost algorithms. Furthermore, the optimal time window length is 3.5 s, with an average F1 of 0.877. This study provides an effective method for real-time driving behavior identification, and thereby supports the driving pattern analysis and Advanced Driving Assistance System.