Mine ecological restoration is a critical process for promoting the sustainable development of resource-dependent regions, yet existing monitoring methods remain limited in accuracy and adaptability. To address challenges such as small-object recognition, insufficient multi-scale feature fusion, and blurred boundaries in UAV-based remote sensing imagery, this paper proposes an enhanced semantic segmentation model based on Segformer. Specifically, a multi-scale feature-enhanced feature pyramid network (MSFE-FPN) is introduced between the encoder and decoder to strengthen cross-level feature interaction. Additionally, a selective feature aggregation pyramid pooling module (SFA-PPM) is integrated into the deepest feature layer to improve global semantic perception, while an efficient local attention (ELA) module is embedded into lateral connections to enhance sensitivity to edge structures and small-scale targets. A high-resolution UAV image dataset, named the HUNAN Mine UAV Dataset (HNMUD), is constructed to evaluate model performance, and further validation is conducted on the public Aeroscapes dataset. Experimental results demonstrated that the proposed method exhibited strong performance in terms of segmentation accuracy and generalization ability, effectively supporting the image analysis needs of mine restoration scenes.