In recent years, with the rapid development of artificial intelligence technology, computer vision-based pest detection technology has been widely used in agricultural production. Tomato diseases and pests are serious problems affecting tomato yield and quality, so it is important to detect them quickly and accurately. In this paper, we propose a tomato disease and pest detection model based on an improved YOLOv5n to overcome the problems of low accuracy and large model size in traditional pest detection methods. Firstly, we use the Efficient Vision Transformer as the feature extraction backbone network to reduce model parameters and computational complexity while improving detection accuracy, thus solving the problems of poor real-time performance and model deployment. Second, we replace the original nearest neighbor interpolation upsampling module with the lightweight general-purpose upsampling operator Content-Aware ReAssembly of FEatures to reduce feature information loss during upsampling. Finally, we use Wise-IoU instead of the original CIoU as the regression loss function of the target bounding box to improve the regression prediction accuracy of the predicted bounding box while accelerating the convergence speed of the regression loss function. We perform statistical analysis on the experimental results of tomato diseases and pests under data augmentation conditions. The results show that the improved algorithm improves mAP50 and mAP50:95 by 2.3% and 1.7%, respectively, while reducing the number of model parameters by 0.4 M and the computational complexity by 0.9 GFLOPs. The improved model has a parameter count of only 1.6 M and a computational complexity of only 3.3 GFLOPs, demonstrating a certain advantage over other mainstream object detection algorithms in terms of detection accuracy, model parameter count, and computational complexity. The experimental results show that this method is suitable for the early detection of tomato diseases and pests.