Recently, disease prevention in jute plants has become an urgent topic as a result of the growing demand for finer quality fiber. This research presents a deep learning network called YOLO-JD for detecting jute diseases from images. In the main architecture of YOLO-JD, we integrated three new modules such as Sand Clock Feature Extraction Module (SCFEM), Deep Sand Clock Feature Extraction Module (DSCFEM), and Spatial Pyramid Pooling Module (SPPM) to extract image features effectively. We also built a new large-scale image dataset for jute diseases and pests with ten classes. Compared with other state-of-the-art experiments, YOLO-JD has achieved the best detection accuracy, with an average mAP of 96.63%.