To investigate the effect of deep neural networks with transfer learning on MR images for tumor classification and improve the classification metrics by building image-level, stratified image-level, and patient-level models. Three thousand sixty-four T1-weighted magnetic resonance (MR) imaging from two hundred thirty-three patient cases of three brain tumors types (meningioma, glioma, and pituitary) were collected and it includes coronal, sagittal and axial views. The average number of brain images of each patient in three views is fourteen in the collected dataset. The classification is performed in a model of cross-trained with a pre-trained InceptionV3 model. Three image-level and one patient-level models are built on the MR imaging dataset. The models are evaluated in classification metrics such as accuracy, loss, precision, recall, kappa, and AUC. The proposed models are validated using four approaches: holdout validation, 10-fold cross-validation, stratified 10-fold cross-validation, and group 10-fold cross-validation. The generalization capability and improvement of the network are tested by using cropped and uncropped images of the dataset. The best results for group 10-fold cross-validation (patient-level) are obtained on the used dataset (ACC=99.82). A deep neural network with transfer learning can be used to classify brain tumors from MR images. Our patient-level network model noted the best results in classification to improve accuracy.