Sunflower is a crop that has many economic values and ornamental usages. However, its production can be hampered due to various diseases such as downy mildew, gray mold, and leaf scars, and it is challenging for farmers to identify disease-prone conditions with traditional approaches. Thus, a computerized model composed of vision, artificial intelligence, and machine learning is the demand of the age to detect diseases in plants efficiently. In this paper, we develop a hybrid model with transfer learning (TL) and a simple CNN using a small dataset for detecting sunflower diseases. Out of the eight models tested on the dataset of four different classes (downy mildew, gray mold, leaf scars, and fresh leaf), the VGG19 + CNN hybrid model achieves the best results in terms of precision, recall, F1-score, accuracy, Hamming loss, Matthews coefficient, Jaccard score, and Cohen’s kappa metrics. The experimental outcomes show that the proposed model provides better precision, recall, and accuracy than other approaches on the benchmark dataset.