In the rapidly evolving field of semiconductor manufacturing, the escalating complexity of integrated circuits poses significant challenges in identifying and analyzing defects, crucial for maintaining high wafer yield. Traditional approaches are hindered by the intricate nature of defect morphology and the scarcity of high-quality scanning electron microscope (SEM) data, essential for effective algorithm training. In this study, we propose RA-UNet for fine-grained SEM defect segmentation. RA-UNet adopts a U-shaped architecture that leverages residual networks for defect feature extraction, and introduces a residual architecture and a novel attention module to enhance the network’s focus on defects. To validate the effectiveness of the proposed model, we meticulously gathered and labeled a real SEM data set from a semiconductor manufacturing factory. The results demonstrate that RA-UNet outperforms existing methods in semiconductor defect segmentation, achieving an Intersection over Union (IoU) score of 71.92%. These findings highlight its potential as an effective tool for semiconductor defect analysis.