This paper proposes a method for vehicle driving behavior recognition (VDBR) based on a six-axis motion processor. This method uses deep learning technology to learn the sample data collected by the on-board sensor. To solve the problem of small sample size and easy overfitting, we propose a joint data augmentation (JDA) scheme, and design a new multi-view convolutional neural network model (MV-CNN). The JDA includes the multi-axis weighted fusion (MAWF) algorithm, background noise fusion (BNF) algorithm and random cropping (RC) algorithm to construct a sample dataset that is more in line with a complex real driving environment. With the CNN model, the direction of information propagation improves, and a new MV-CNN model was developed for the training, learning and recognition of driving behavior. The performance of MV-CNN is experimentally compared with CNN, RNN, LSTM, CNN+LSTM and 3D-CNN. The results show that MV-CNN can obtain the best recall, precision, and F1-score. At the same time, MV-CNN and JDA have better generalization ability, reduce the training variance and deviation, and increase the stability of the model training process.