Specular highlight detection is a fundamental research topic in computer graphics and computer vision. In this paper, we present a new full-scale deep supervision model to detect specular highlights from single real-world images. The core of our approach is a novel self-attention module to improve the detection accuracy of the network. We also introduce a refinement strategy with a new loss function for highlight detection task by generating contour maps from the highlight detection masks. Experiments on a public dataset demonstrate that our approach outperforms state-of-the-art methods for highlight detection.