To address the challenges of low accuracy in coal gangue image recognition and poor segmentation performance under the influence of dust in underground coal mines, a scaled simulation platform was constructed to replicate the longwall top coal caving face. This platform utilized real coal gangue particles as the raw material and employed dust simulation to mimic the dust conditions typically found in coal mines. Images of coal gangue without dust and under varying dust concentrations were then collected for analysis. In parallel, an improved DeeplabV3+ coal gangue image segmentation model is proposed, where ResNeSt is employed as the backbone network of DeeplabV3+, thereby enhancing the model’s capability to extract features of both coal and gangue. Furthermore, two channel attention modules (ECAs) are incorporated to augment the model’s ability to recognize edge features in coal gangue images. A class-label smoothing training strategy was adopted for model training. The experimental results indicate that, compared to the original DeepLabV3+ model, the optimized model achieves improvements of 3.14%, 4.70%, and 3.83% in average accuracy, mean intersection over union (mIoU), and mean pixel accuracy, respectively. Furthermore, the number of parameters was reduced from 44.18 M to 43.86 M, the floating-point operations decreased by 8.33%, and the frames per second (FPS) increased by 45.03%. When compared to other models such as UNet, PSANet, and SegFormer, the proposed model demonstrates superior performance in coal gangue segmentation, accuracy, and parameter efficiency. A method combining dark channel prior and Gaussian weighting was employed for defogging coal gangue images under varying dust concentration conditions. The recognition performance of the coal gangue images before and after defogging was assessed across different dust concentrations. The model’s segmentation accuracy and practical applicability were validated through defogging and segmentation of both indoor and underground dust images. The recognition accuracy of coal and gangue, before and after defogging, improved by 6.8–71.8% and 5.8–45.8%, respectively, as the dust concentration increased, thereby demonstrating the model’s effectiveness in coal gangue image defogging segmentation in underground dust environments.