Cotton diseases and pests pose significant threats to cotton production, necessitating accurate and efficient classification methods. Despite existing advanced methods, there is a research gap in utilizing both local feature extraction and global context capture for enhanced classification accuracy. Hence, this study developed and evaluated three advanced models for cotton disease and pest classification: a convolutional neural network (CNN)-based model, a Vision Transformer (ViT)-based model, and a hybrid CNN-ViT model. These models were trained on a dataset comprising eight classes of cotton diseases and pests, namely aphids, armyworm, bacterial blight, cotton boll rot, green cotton boll, healthy, powdery mildew, and target spot. The results demonstrated that the hybrid CNN-ViT model achieved the highest overall performance with an average test accuracy of 98.5%. The CNN model showed strong performance with an average accuracy of 97.9%. The ViT models, while having self-attention mechanisms to capture context and dependencies, exhibited improved performance with increased depth. The ViT model having four transformer layers outperformed the two-layer variant, achieving an average accuracy of 97.2% compared to 96.3%. The hybrid model effectively combined the strengths of CNN's local feature extraction and ViT's global feature capture, resulting in superior classification accuracy across most classes. Future research should focus on expanding the dataset to include more diverse diseases and pests and integrating the models with autonomous platforms for spraying the chemicals, thus facilitating real-world adoption and application in agricultural settings.