In recent years, computer vision technology has been widely used in the field of medical image processing. However, there is still a big gap between the existing breast mass detection methods and the real-world application due to the limited detection accuracy. It is known that humans locate the regions of interest quickly and further identify whether these regions are the targets we found. In breast cancer diagnosis, we locate all the potential regions of breast mass by glancing at the mammographic image from top to bottom and from left to right, then further identify whether these regions are a breast mass. Inspired by the process of human detection of breast mass, we proposed a novel breast mass detection method to detect breast mass on a mammographic image by stimulating the process of human detection. The proposed method preprocesses the mammographic image via the mathematical morphology method and locates the suspected regions of breast mass by the image template matching method. Then, it obtains the regions of breast mass by classifying these suspected regions into breast mass and background categories using a convolutional neural network (CNN). The bounding box of breast mass obtained by the mathematical morphology method and image template matching method are roughly due to the mathematical morphology method, which transforms all of the brighter regions into approximate circular areas. For regression of a breast mass bounding box, the optimal solution should be searched in the feasible region and the Particle Swarm Optimization (PSO) is suitable for solving the problem of searching the optimal solution within a certain range. Therefore, we refine the bounding box of breast mass by the PSO algorithm. The proposed breast mass detection method and the compared detection methods were evaluated on the open database Digital Database for Screening Mammography (DDSM). The experimental results demonstrate that the proposed method is superior to all of the compared detection methods in detection performance.