Wheat Stripe Rust Disease (WRD) poses a significant threat to wheat crops, causing substantial yield losses and can result in total crop damage if not detected early. The localization of WRD-infected areas is a labor-intensive and time-consuming task due to the intricate and varied nature of the disease spread, especially for large plantations. Hence, segmentation of wheat crops is vital for early identification of the WRD-affected area, which allows for the implementation of targeted intervention measures. The state-of-the-art segmentation technique for WRD using the real-world semantic segmentation NWRD dataset is based on a UNet model with the Adaptive Patching with Feedback (APF) technique. However, this implementation is complex and requires significant resources and time for training due to the processing of each patch of the dataset. Our work in this paper improves the state-of-the-art by using a two-stage model: a Vision Transformer (ViT) classifier to distinguish between the rust and non-rust patches and a less complex co-salient object detection (Co-SOD) model for segmentation of the classified images. The Co-SOD model uses multiple rust patches to extract contextual features from a group of images. By analyzing multiple patches of wheat rust disease simultaneously, we can segment disease regions more accurately. Our results show that the proposed approach achieves a higher F1 score (0.638), precision (0.621), and recall (0.675) for the rust class with $5\times$ less training time as compared to the previous works.