In robot-assisted surgical systems, surgical instrument segmentation is a critical task that provides important information for surgeons to make informed decisions and ensure surgical safety. However, current mainstream models often lack precise segmentation edges and suffer from an excess of parameters, rendering their deployment challenging. To address these issues, this article proposes a lightweight semantic segmentation model based on edge refinement and efficient self-attention. The proposed model utilizes a lightweight densely connected network for feature extraction, which is able to extract high-quality semantic information with fewer parameters. The decoder combines a feature pyramid module with an efficient criss-cross self-attention module. This fusion integrates multi-scale data, strengthens focus on surgical instrument details, and enhances edge segmentation accuracy. To train and evaluate the proposed model, the authors developed a private dataset of endoscopic surgical instruments. It containing 1,406 images for training, 469 images for validation and 469 images for testing. The proposed model performs well on this dataset with only 466 K parameters, achieving a mean Intersection over Union (mIoU) of 97.11%. In addition, the model was trained on public datasets Kvasir-instrument and Endovis2017. Excellent results of 93.24% and 95.83% were achieved on the indicator mIoU, respectively. The superiority and effectiveness of the method are proved. Experimental results show that the proposed model has lower parameters and higher accuracy than other state-of-the-art models. The proposed model thus lays the foundation for further research in the field of surgical instrument segmentation.