Mass detection in mammograms has a limited approach to the presence of a mass in overlapping denser fibroglandular breast regions. In addition, various breast density levels could decrease the learning system’s ability to extract sufficient feature descriptors and may result in lower accuracy performance. Therefore, this study is proposing a textural-based image enhancement technique named Spatial-based Breast Density Enhancement for Mass Detection (SbBDEM) to boost textural features of the overlapped mass region based on the breast density level. This approach determines the optimal exposure threshold of the images’ lower contrast limit and optimizes the parameters by selecting the best intensity factor guided by the best Blind/Reference-less Image Spatial Quality Evaluator (BRISQUE) scores separately for both dense and non-dense breast classes prior to training. Meanwhile, a modified You Only Look Once v3 (YOLOv3) architecture is employed for mass detection by specifically assigning an extra number of higher-valued anchor boxes to the shallower detection head using the enhanced image. The experimental results show that the use of SbBDEM prior to training mass detection promotes superior performance with an increase in mean Average Precision (mAP) of 17.24% improvement over the non-enhanced trained image for mass detection, mass segmentation of 94.41% accuracy, and 96% accuracy for benign and malignant mass classification. Enhancing the mammogram images based on breast density is proven to increase the overall system’s performance and can aid in an improved clinical diagnosis process.

The convolutional neural network (CNN) has been explored for mammogram cancer classification to aid radiologists. CNNs require multiple convolution and non-linearity repetitions to learn data sparsity, but deeper networks often face the vanishing gradient effect, which hinders effective learning. The rectified linear unit (ReLU) activation function activates neurons only when the output exceeds zero, limiting activation and potentially lowering performance. This study proposes an adaptive ReLU based on a genetic algorithm (GA) to determine the optimal threshold for neuron activation, thus improving the restrictive nature of the original ReLU. We compared performances on the INbreast and IPPT-mammo mammogram datasets using ReLU and leakyReLU activation functions. Results show accuracy improvements from 95.0% to 97.01% for INbreast and 84.9% to 87.4% for IPPT-mammo with ReLU and from 93.03% to 99.0% for INbreast and 84.03% to 91.06% for IPPT-mammo with leakyReLU. Significant accuracy improvements were observed for breast cancer classification in mammograms, demonstrating its potential to aid radiologists with more robust and reliable diagnostic tools.