The world-wide cultivation of dragon fruit is expanding rapidly; however, its susceptibility to diseases poses a huge threat to the yield and profitability. Therefore, in order to resolve such problems, research continues in the area of automating disease recognition using advanced deep learning methods. Likewise, this research suggests a very effective method for accurate classification of dragon fruit plant stem diseases, which is essential to ensure better crop yields and lower agricultural waste. Different machine learning models like KNearest Neighbors, Convolutional Neural Networks, Support Vector Machine, VGG16, ResNet, DenseNet, EfficientNet, and Vision Transformer (ViT) have been tested for disease classification. Amongst them, the ViT model achieves the best accuracy of 87.18% in disease detection and proves to be very efficient for computer-aided plant disease diagnosis.