Ovarian cancer is the most common cause of death among gynecological malignancies, due to its multiple subtypes that pose significant diagnostic challenges. In general, improving classification accuracy leads to earlier diagnosis and more successful treatment. This study presents a new methodology based on the DenseNet121 model architecture with the integration of the GAM Attention module, which combines ductal and spatial attention mechanisms. This module aims to improve the model’s ability to focus on the most relevant regions and features in ovarian MRI images, thereby improving classification accuracy. Additionally, a Bayesian optimization algorithm is used to find the best set of hyperparameters to train the proposed model. The performance of the proposed model was tested on the KAUH-OCM ovarian cancer MRI dataset obtained from King Abdullah University Hospital (KAUH). The results demonstrated that the proposed model outperformed previously trained models (DenseNet121, VGG16, VGG19, ResNet50, and InceptionV3) with an accuracy of 91.84%, a specificity of 96.76%, and an F1 score of 91.78%. Furthermore, the proposed model was also tested on the KAUH-OCCTD ovarian cancer CT dataset and the polycystic ovary syndrome (PCOS) ultrasound dataset, achieving high accuracy rates of 97.27% and 100%, respectively. These results demonstrate the potential of the proposed model to revolutionize ovarian cancer diagnosis with high accuracy. These findings may guide gynecologists in providing better treatment strategies for these patients.