Brain tumors pose significant risks to cognitive functions, making early detection crucial for improving patient survival rates. Accurate tumor detection aids neuro-oncologists in diagnosing tumor types and recommending appropriate treatments. However, manual detection is challenging, time-consuming, and prone to human error. Recently, Deep Learning (DL) models have demonstrated substantial potential in efficiently classifying large image datasets. This paper presents three novel and efficient multi-class DL architectures leveraging transfer learning to classify different brain tumors. Our primary contribution is to enhance the predictive accuracy while minimizing the usage of computational resource compared to other state-of-art models in the literature by forgoing preprocessing, segmentation, hybrid models, and augmentation techniques. Additionally, we reduce the number of FC layers to streamline computation. We conduct extensive experiments to evaluate the performance of our models using the brain tumor Figshare T1-weighted contrast-enhanced MRI dataset, comprising 3064 images from three distinct tumor types. The InceptionV3 model records a 98.36% accuracy level with five-fold cross-validation. By incorporating batch normalization and optimizing the learning rate for the Adam optimizer, the Xception model reached 99.19% accuracy. Finally, we utilize Particle Swarm Optimization (PSO) to fine-tune the learning rate of the Stochastic Gradient Descent (SGD) optimizer. The Xception model attained 98.85% accuracy. These results highlight the novelty of our approach, offering a practical solution for neuro-oncologists, particularly through the fine-tuned Xception model, which delivers early and accurate tumor detection with minimal computational resources.