Cotton is one of the world’s most economically significant agricultural products; however, it is susceptible to numerous pest and virus attacks during the growing season. Pests (whitefly) can significantly affect a cotton crop, but timely disease detection can help pest control. Deep learning models are best suited for plant disease classification. However, data scarcity remains a critical bottleneck for rapidly growing computer vision applications. Several deep learning models have demonstrated remarkable results in disease classification. However, these models have been trained on small datasets that are not reliable due to model generalization issues. In this study, we first developed a dataset on whitefly attacked leaves containing 5135 images that are divided into two main classes, namely, (i) healthy and (ii) unhealthy. Subsequently, we proposed a Compact Convolutional Transformer (CCT)-based approach to classify the image dataset. Experimental results demonstrate the proposed CCT-based approach’s effectiveness compared to the state-of-the-art approaches. Our proposed model achieved an accuracy of 97.2%, whereas Mobile Net, ResNet152v2, and VGG-16 achieved accuracies of 95%, 92%, and 90%, respectively.