Traditional pest and disease management methods are inefficient, relying on agricultural experts or static resources, making it difficult to respond quickly to large-scale outbreaks and meet local needs. Although deep learning technologies have been applied in pest and disease management, challenges remain, such as the dependence on large amounts of manually labeled data and the limitations of dynamic reasoning. To address these challenges, this study proposes IPM-AgriGPT (Integrated Pest Management—Agricultural Generative Pre-Trained Transformer), a Chinese large language model specifically designed for pest and disease knowledge. The proposed Generation-Evaluation Adversarial (G-EA) framework is used to generate high-quality question–answer corpora and combined with Agricultural Contextual Reasoning Chain-of-Thought Distillation (ACR-CoTD) and low-rank adaptation (LoRA) techniques further optimizes the base model to build IPM-AgriGPT. During the evaluation phase, this study designed a specialized benchmark for the agricultural pest and disease domain, comprehensively assessing the performance of IPM-AgriGPT in pest management tasks. Experimental results show that IPM-AgriGPT achieved excellent evaluation scores in multiple tasks, demonstrating its great potential in agricultural intelligence and pest management.